Nov 02 13:40:43.990364 ip-10-0-140-236 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 13:40:43.990372 ip-10-0-140-236 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 13:40:43.990379 ip-10-0-140-236 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 13:40:43.990604 ip-10-0-140-236 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 13:40:54.084903 ip-10-0-140-236 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 13:40:54.084918 ip-10-0-140-236 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8672f83269ed4b4b96cedec1f4607134 -- Nov 02 13:43:00.084271 ip-10-0-140-236 systemd[1]: Starting Kubernetes Kubelet... Nov 02 13:43:00.495112 ip-10-0-140-236 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:43:00.495112 ip-10-0-140-236 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 13:43:00.495112 ip-10-0-140-236 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:43:00.495112 ip-10-0-140-236 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:43:00.501439 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501353 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:43:00.501439 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501437 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:43:00.501439 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501441 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501445 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501448 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501451 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501454 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501456 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501459 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501462 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501465 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501468 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501470 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501473 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501476 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501478 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501481 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501483 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501486 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501489 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501492 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501494 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:43:00.501538 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501503 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501506 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501509 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501511 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501514 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501517 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501519 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501522 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501525 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501527 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501530 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501532 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501535 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501539 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501542 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501545 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501548 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501552 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501554 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:43:00.502022 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501557 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501559 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501562 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501566 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501571 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501574 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501577 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501580 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501582 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501586 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501590 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501594 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501597 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501600 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501603 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501605 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501608 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501611 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501613 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:43:00.502510 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501616 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501618 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501621 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501624 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501626 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501629 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501632 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501637 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501639 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501642 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501645 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501647 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501650 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501653 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501655 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501658 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501661 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501664 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501667 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501670 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:43:00.502964 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501672 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501676 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501678 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501681 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501683 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501686 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501689 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501691 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501694 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501696 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501699 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501702 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501704 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501707 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501710 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501713 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501715 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501719 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501722 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:43:00.503484 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501725 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501728 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501731 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501734 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501736 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501739 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.501742 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502544 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502551 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502554 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502557 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502561 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502563 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502567 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502570 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502572 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502575 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502578 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502581 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:43:00.503938 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502583 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502586 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502589 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502592 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502595 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502597 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502600 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502603 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502605 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502609 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502611 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502615 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502619 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502624 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502627 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502630 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502633 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502637 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502639 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:43:00.504411 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502642 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502645 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502648 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502650 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502653 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502656 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502658 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502661 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502669 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502672 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502675 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502678 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502680 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502686 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502689 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502691 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502695 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502697 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502700 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502703 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:43:00.504872 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502705 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502709 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502712 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502714 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502717 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502720 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502722 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502725 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502728 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502731 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502734 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502736 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502739 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502742 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502745 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502747 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502750 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502753 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502755 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:43:00.505373 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502758 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502761 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502764 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502768 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502772 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502775 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502778 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502780 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502783 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502786 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502789 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502792 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502795 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502797 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502802 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502805 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502808 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502811 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502813 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502816 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:43:00.505852 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502819 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502821 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502824 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502827 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502829 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502833 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502835 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502838 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502841 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502843 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502846 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502848 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502851 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502854 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502856 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.502860 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503361 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503366 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503369 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503372 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:43:00.506343 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503375 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503378 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503381 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503386 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503389 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503392 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503396 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503398 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503401 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503404 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503407 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503409 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503412 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503415 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503419 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503423 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503428 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503431 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503433 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:43:00.506819 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503436 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503438 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503441 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503444 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503446 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503449 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503452 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503454 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503458 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503460 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503463 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503466 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503468 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503472 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503474 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503477 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503479 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503482 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503484 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503488 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:43:00.507288 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503491 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503494 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503497 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503500 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503502 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503505 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503508 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503510 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503513 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503517 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503520 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503523 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503525 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503528 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503531 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503534 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503536 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503539 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503541 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:43:00.507762 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503544 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503547 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503551 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503553 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503556 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503558 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503561 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503564 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503567 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503569 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503572 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503575 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503578 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503581 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503584 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503587 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503589 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503592 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503595 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:43:00.508219 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503597 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503600 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503603 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503607 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503610 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503612 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503615 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503618 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503620 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503623 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503625 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503628 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503631 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503633 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503636 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503639 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503642 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503645 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503647 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503650 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:43:00.508680 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503653 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503655 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503658 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503660 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.503664 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504823 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504838 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504845 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504850 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504855 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504859 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504864 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504868 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504871 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504874 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504879 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504882 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504886 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504889 2579 flags.go:64] FLAG: --cgroup-root="" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504892 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504895 2579 flags.go:64] FLAG: --client-ca-file="" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504898 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504901 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504905 2579 flags.go:64] FLAG: --cluster-domain="" Nov 02 13:43:00.509150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504908 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504912 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504915 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504918 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504922 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504927 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504930 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504933 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504937 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504940 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504943 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504946 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504952 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504956 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504959 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504963 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504966 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504969 2579 flags.go:64] FLAG: --enable-server="true" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504972 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504976 2579 flags.go:64] FLAG: --event-burst="100" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504991 2579 flags.go:64] FLAG: --event-qps="50" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504995 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.504998 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505001 2579 flags.go:64] FLAG: --eviction-hard="" Nov 02 13:43:00.509795 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505005 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505008 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505012 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505015 2579 flags.go:64] FLAG: --eviction-soft="" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505018 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505021 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505024 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505027 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505030 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505033 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505036 2579 flags.go:64] FLAG: --feature-gates="" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505040 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505044 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505047 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505050 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505054 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505057 2579 flags.go:64] FLAG: --help="false" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505060 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505064 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505067 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505071 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505075 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505078 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505081 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 13:43:00.510432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505084 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505087 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505090 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505093 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505096 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505099 2579 flags.go:64] FLAG: --kube-reserved="" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505102 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505105 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505108 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505111 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505114 2579 flags.go:64] FLAG: --lock-file="" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505117 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505121 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505124 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505130 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505133 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505136 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505139 2579 flags.go:64] FLAG: --logging-format="text" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505142 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505145 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505148 2579 flags.go:64] FLAG: --manifest-url="" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505151 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505156 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505160 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505163 2579 flags.go:64] FLAG: --max-pods="110" Nov 02 13:43:00.511021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505167 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505170 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505173 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505177 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505180 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505183 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505186 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505193 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505196 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505199 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505202 2579 flags.go:64] FLAG: --pod-cidr="" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505205 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505209 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505212 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505215 2579 flags.go:64] FLAG: --port="10250" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505218 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505221 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ae846ada92c766e9" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505224 2579 flags.go:64] FLAG: --qos-reserved="" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505227 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505230 2579 flags.go:64] FLAG: --register-node="true" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505233 2579 flags.go:64] FLAG: --register-with-taints="" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505237 2579 flags.go:64] FLAG: --registry-burst="10" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505240 2579 flags.go:64] FLAG: --registry-qps="5" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505243 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505246 2579 flags.go:64] FLAG: --reserved-memory="" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505250 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 13:43:00.511611 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505253 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505256 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505259 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505262 2579 flags.go:64] FLAG: --runonce="false" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505265 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505268 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505271 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505274 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505277 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505281 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505285 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505288 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505291 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505294 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505297 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505300 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505303 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505306 2579 flags.go:64] FLAG: --system-cgroups="" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505310 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505316 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505319 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505322 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505326 2579 flags.go:64] FLAG: --tls-min-version="" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505329 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505332 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 13:43:00.512264 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505335 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505338 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505341 2579 flags.go:64] FLAG: --v="2" Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505351 2579 flags.go:64] FLAG: --version="false" Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505355 2579 flags.go:64] FLAG: --vmodule="" Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505359 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505362 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505495 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505499 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505502 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505505 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505508 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505511 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505514 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505518 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505523 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505528 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505531 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505534 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505537 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:43:00.512890 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505539 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505542 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505545 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505548 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505550 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505554 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505557 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505560 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505563 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505566 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505569 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505571 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505574 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505577 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505580 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505582 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505585 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505588 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505590 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505593 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:43:00.513381 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505596 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505598 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505601 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505604 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505606 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505609 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505612 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505615 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505619 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505622 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505625 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505627 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505630 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505633 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505636 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505639 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505642 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505646 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505649 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505652 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:43:00.513853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505654 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505657 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505660 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505662 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505665 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505668 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505671 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505674 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505677 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505680 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505682 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505685 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505688 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505691 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505693 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505696 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505698 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505701 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505704 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505707 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:43:00.514354 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505711 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505713 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505716 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505718 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505721 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505724 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505727 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505729 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505732 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505736 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505739 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505741 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505744 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505747 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505749 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505752 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505755 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505757 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505760 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505763 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:43:00.514871 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505765 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505768 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505771 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505773 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505776 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505779 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505782 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505784 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505788 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505792 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505795 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505798 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505803 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.505830 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505865 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505870 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:43:00.515361 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505873 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505877 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505880 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505882 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505885 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505889 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505892 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505895 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505898 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505900 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505903 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505906 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505909 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505911 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505915 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505918 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505920 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505923 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505925 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505928 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:43:00.515751 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505931 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505934 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505936 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505939 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505941 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505944 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505947 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505950 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505952 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505956 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505959 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505961 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505964 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505967 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505969 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505972 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505974 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505978 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505992 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:43:00.516237 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505995 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.505997 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506000 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506003 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506005 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506008 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506010 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506013 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506016 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506019 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506021 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506024 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506026 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506029 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506031 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506034 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506037 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506039 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506042 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506044 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:43:00.516682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506047 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506050 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506052 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506063 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506066 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506068 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506071 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506074 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506076 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506079 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506083 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506086 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506088 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506091 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506095 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506099 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506103 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506107 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506110 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:43:00.517163 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506113 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506115 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506118 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506121 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506123 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506126 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506128 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506132 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506136 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506140 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506143 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506145 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506148 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506151 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506153 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506156 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506160 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506163 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506166 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:43:00.517608 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506168 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506171 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506173 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506176 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506180 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506182 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.506185 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.513969 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.514000 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.514017 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.514021 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514079 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514084 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514088 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514093 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514096 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514099 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514102 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514105 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514107 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:43:00.518079 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514110 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514113 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514115 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514118 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514121 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514124 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514127 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514130 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514132 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514135 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514139 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514142 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514144 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514147 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514150 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514153 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514155 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514158 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514161 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:43:00.518572 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514164 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514167 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514170 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514173 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514175 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514178 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514181 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514184 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514187 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514190 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514192 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514195 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514197 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514200 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514203 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514205 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514208 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514211 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514213 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514216 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:43:00.519048 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514219 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514221 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514224 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514227 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514230 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514234 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514238 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514241 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514244 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514246 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514250 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514253 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514255 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514259 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514262 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514264 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514267 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514269 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514272 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:43:00.519522 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514275 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514278 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514280 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514283 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514286 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514288 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514291 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514294 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514296 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514299 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514301 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514304 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514308 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514311 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514314 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514317 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514320 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514322 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514330 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514333 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:43:00.519990 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514335 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514338 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514341 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514344 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514347 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514350 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514353 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514356 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514359 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514362 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514365 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514368 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514370 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514373 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514376 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514379 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514382 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514385 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:43:00.520506 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514387 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.514416 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514552 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514556 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514559 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514562 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514565 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514568 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514571 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514573 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514577 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514582 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514585 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514588 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514591 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514594 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:43:00.521202 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514597 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514599 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514602 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514605 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514609 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514612 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514614 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514617 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514620 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514622 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514625 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514628 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514631 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514633 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514636 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514639 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514642 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514644 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514647 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514649 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:43:00.521632 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514652 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514655 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514657 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514660 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514662 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514665 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514669 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514673 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514676 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514678 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514682 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514684 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514687 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514690 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514693 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514696 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514698 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514701 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514703 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:43:00.522159 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514706 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514709 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514711 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514714 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514717 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514719 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514722 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514724 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514727 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514729 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514732 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514735 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514738 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514740 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514743 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514746 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514748 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514751 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514753 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514756 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:43:00.522785 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514758 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514761 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514763 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514766 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514769 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514772 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514774 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514778 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514780 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514783 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514785 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514788 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514790 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514793 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514796 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514799 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514801 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514804 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514807 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514810 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:43:00.523461 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514812 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514814 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514817 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514820 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514822 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514825 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514827 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514830 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514833 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514835 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514838 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514840 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514843 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.514865 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514897 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:43:00.523948 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514901 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514904 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514908 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514910 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514914 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514916 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514919 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514923 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514927 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514929 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514932 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514935 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514938 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514941 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514943 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514946 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514949 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514951 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514954 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:43:00.524334 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514959 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514961 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514964 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514967 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514969 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514972 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514974 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514977 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514993 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514997 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.514999 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515002 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515005 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515007 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515010 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515013 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515016 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515019 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:43:00.524802 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515021 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515025 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515029 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515032 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515035 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515038 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515042 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515045 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515047 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515050 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515053 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515055 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515058 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515061 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515063 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515066 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515069 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515071 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515074 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:43:00.525265 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515076 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515079 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515081 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515084 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515087 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515089 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515092 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515094 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515097 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515100 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515103 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515106 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515109 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515111 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515114 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515117 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515120 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515122 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515125 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515128 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:43:00.525744 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515130 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515133 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515135 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515138 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515141 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515143 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515146 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515148 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515151 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515154 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515157 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515159 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515162 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515164 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515167 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515170 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515172 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515175 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515177 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:43:00.526255 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515180 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515182 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515185 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515188 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515191 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515195 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515198 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515200 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:00.515203 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.515252 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.517947 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.518822 2579 server.go:1014] "Starting client certificate rotation" Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.518912 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.519592 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:43:00.526693 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.520865 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:43:00.527974 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.527953 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:43:00.538806 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.538787 2579 log.go:25] "Validated CRI v1 runtime API" Nov 02 13:43:00.542060 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.542044 2579 log.go:25] "Validated CRI v1 image API" Nov 02 13:43:00.542892 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.542866 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 13:43:00.545582 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.545562 2579 fs.go:135] Filesystem UUIDs: map[13807ce8-e841-409d-97c5-2405b69749da:/dev/nvme0n1p3 4f775794-7b44-418a-a8a5-312c5dc5500f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 02 13:43:00.545629 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.545582 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 13:43:00.550732 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.550616 2579 manager.go:217] Machine: {Timestamp:2026-11-02 13:43:00.54929737 +0000 UTC m=+0.344732782 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098602 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec222168b52a39c866fc9f10df963976 SystemUUID:ec222168-b52a-39c8-66fc-9f10df963976 BootID:8672f832-69ed-4b4b-96ce-dec1f4607134 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:bb:0f:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:bb:0f:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:68:a9:a1:c0:bb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 13:43:00.550821 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.550734 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 13:43:00.550857 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.550842 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 13:43:00.553692 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.553664 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 13:43:00.553827 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.553694 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 13:43:00.553867 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.553829 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 13:43:00.553867 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.553837 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 13:43:00.553944 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.553870 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:43:00.553944 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.553888 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:43:00.554716 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.554705 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 13:43:00.558143 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.558131 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 13:43:00.558259 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.558250 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 13:43:00.559164 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.559155 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 02 13:43:00.559195 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.559168 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 13:43:00.559195 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.559178 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 13:43:00.559195 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.559188 2579 kubelet.go:405] "Adding apiserver pod source" Nov 02 13:43:00.559303 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.559197 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 13:43:00.559521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.559509 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:43:00.562093 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.562065 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 13:43:00.567086 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.567073 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:43:00.568099 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568083 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 13:43:00.568152 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568116 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 13:43:00.568276 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568265 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 13:43:00.568310 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568287 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 13:43:00.568310 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568297 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 13:43:00.568310 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568307 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 13:43:00.568399 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568316 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 13:43:00.568399 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568327 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 13:43:00.568972 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568962 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 13:43:00.569028 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568972 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 13:43:00.569028 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.568995 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 13:43:00.569028 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.569004 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 13:43:00.569028 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.569013 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 13:43:00.569028 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.569022 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 13:43:00.571077 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.571068 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 13:43:00.571077 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.571077 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 13:43:00.574810 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.574798 2579 server.go:1263] "Started kubelet" Nov 02 13:43:00.574899 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.574875 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 13:43:00.580394 ip-10-0-140-236 systemd[1]: Started Kubernetes Kubelet. Nov 02 13:43:00.580601 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.579613 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 13:43:00.580665 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.580606 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 13:43:00.582887 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.582224 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 13:43:00.585249 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.585226 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 02 13:43:00.588224 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.588201 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 13:43:00.590397 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.590380 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 13:43:00.591399 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.591382 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 13:43:00.591703 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.591502 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:00.592005 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.591970 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 13:43:00.592085 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592015 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 13:43:00.592085 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.591970 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 13:43:00.592191 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592117 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 02 13:43:00.592191 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592126 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 02 13:43:00.592282 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592228 2579 factory.go:55] Registering systemd factory Nov 02 13:43:00.592282 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592263 2579 factory.go:223] Registration of the systemd container factory successfully Nov 02 13:43:00.592473 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592452 2579 factory.go:153] Registering CRI-O factory Nov 02 13:43:00.592536 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592477 2579 factory.go:223] Registration of the crio container factory successfully Nov 02 13:43:00.592576 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592540 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 13:43:00.592576 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592561 2579 factory.go:103] Registering Raw factory Nov 02 13:43:00.592654 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.592577 2579 manager.go:1199] Started watching for new ooms in manager Nov 02 13:43:00.593433 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.593413 2579 manager.go:319] Starting recovery of all containers Nov 02 13:43:00.606930 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.606912 2579 manager.go:324] Recovery completed Nov 02 13:43:00.611322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.611307 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:43:00.613287 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.613261 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:43:00.613378 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.613299 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:43:00.613378 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.613318 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:43:00.613979 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.613962 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:43:00.613979 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.613967 2579 cpu_manager.go:226] "Starting" policy="none" Nov 02 13:43:00.614199 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.614011 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 13:43:00.614199 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.614033 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 13:43:00.614847 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.614826 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:43:00.614908 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.614873 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:43:00.614908 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.614889 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:43:00.615292 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.615272 2579 policy_none.go:50] "Start" Nov 02 13:43:00.615292 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.615289 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 13:43:00.615425 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.615303 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 13:43:00.617136 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.617107 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 13:43:00.617329 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.617306 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 13:43:00.617749 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.617727 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 13:43:00.617941 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.616878 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-236.ec2.internal.18e43e3c5cb73706 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-236.ec2.internal,},FirstTimestamp:2026-11-02 13:43:00.57477095 +0000 UTC m=+0.370206362,LastTimestamp:2026-11-02 13:43:00.57477095 +0000 UTC m=+0.370206362,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-236.ec2.internal,}" Nov 02 13:43:00.618059 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.618012 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56mdb" Nov 02 13:43:00.618701 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.618683 2579 policy_none.go:44] "Start" Nov 02 13:43:00.628645 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.628625 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56mdb" Nov 02 13:43:00.637256 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.637224 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 13:43:00.637256 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.637190 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-236.ec2.internal.18e43e3c5f02e0eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-236.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-236.ec2.internal,},FirstTimestamp:2026-11-02 13:43:00.613284075 +0000 UTC m=+0.408719487,LastTimestamp:2026-11-02 13:43:00.613284075 +0000 UTC m=+0.408719487,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-236.ec2.internal,}" Nov 02 13:43:00.637403 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.637294 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 13:43:00.654067 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.654030 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 13:43:00.657476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.657462 2579 manager.go:341] "Starting Device Plugin manager" Nov 02 13:43:00.657558 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.657491 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 13:43:00.657558 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.657500 2579 server.go:87] "Starting device plugin registration server" Nov 02 13:43:00.657755 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.657743 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 13:43:00.657788 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.657756 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 13:43:00.657892 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.657868 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 13:43:00.658077 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.658064 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 13:43:00.658077 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.658077 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 13:43:00.658650 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.658628 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 13:43:00.658717 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.658679 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:00.670262 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.670239 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:43:00.716721 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.716686 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 13:43:00.717941 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.717919 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 13:43:00.718030 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.717944 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 13:43:00.718030 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.717978 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 13:43:00.718143 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.718030 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 13:43:00.734138 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.734116 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:43:00.758748 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.758702 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:43:00.759590 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.759571 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:43:00.759660 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.759603 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:43:00.759660 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.759614 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:43:00.759660 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.759637 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.786411 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.786387 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.786519 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.786415 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-236.ec2.internal\": node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:00.818098 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.818075 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"] Nov 02 13:43:00.818186 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.818128 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:43:00.819644 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.819626 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:43:00.819724 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.819655 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:43:00.819724 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.819669 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:43:00.821021 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821006 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:43:00.821131 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.821166 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821150 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:43:00.821778 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821759 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:43:00.821871 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821791 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:43:00.821871 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821800 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:43:00.821871 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821807 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:43:00.821871 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821817 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:43:00.821871 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.821830 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:43:00.823780 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.823765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.823850 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.823789 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:43:00.824497 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.824480 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:43:00.824604 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.824509 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:43:00.824604 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.824520 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:43:00.842291 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.842270 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-236.ec2.internal\" not found" node="ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.843726 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.843707 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:00.845889 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.845875 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-236.ec2.internal\" not found" node="ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.893432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.893409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.893534 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.893440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.893534 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.893459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.944304 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:00.944281 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:00.993547 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.993518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.993683 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.993551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.993683 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.993586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.993683 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.993631 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.993683 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.993631 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:43:00.993683 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:00.993632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:01.044957 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.044876 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:01.145287 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.145246 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:01.145287 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.145257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:01.149253 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.149020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:43:01.245651 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.245605 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:01.349421 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.349349 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:01.453209 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.453188 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:43:01.491943 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.491922 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:43:01.518607 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.518582 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 13:43:01.519077 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.518722 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 13:43:01.519077 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.518759 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 13:43:01.519077 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.518766 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 13:43:01.560091 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.560068 2579 apiserver.go:52] "Watching apiserver" Nov 02 13:43:01.591040 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.591016 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 13:43:01.591593 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.591578 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:43:01.627222 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.627188 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 13:43:01.627635 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.627611 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-9d7n9","openshift-multus/network-metrics-daemon-v8n7n","openshift-multus/multus-additional-cni-plugins-vbv78","openshift-network-diagnostics/network-check-target-cc5c8","kube-system/konnectivity-agent-8hhsr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4","openshift-network-operator/iptables-alerter-ff8sv","openshift-ovn-kubernetes/ovnkube-node-mq8vn","kube-system/global-pull-secret-syncer-scx6x","openshift-cluster-node-tuning-operator/tuned-ttzxz","openshift-dns/node-resolver-j9lm6","openshift-multus/multus-g268n"] Nov 02 13:43:01.629770 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.629742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.631012 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.630988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:01.631132 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.631096 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:01.632699 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.632362 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:38:00 +0000 UTC" deadline="2028-05-30 05:25:21.346803458 +0000 UTC" Nov 02 13:43:01.632699 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.632415 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13791h42m19.714392707s" Nov 02 13:43:01.632699 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.632426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.634369 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.634346 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:01.634467 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.634419 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:01.635674 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.635655 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:01.637201 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.637178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.638897 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.638480 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:43:01.638897 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.638561 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.638897 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.638574 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:43:01.638897 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.638728 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:43:01.639852 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.639830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.641954 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.641933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g268n" Nov 02 13:43:01.643516 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.643493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.643605 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.643564 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:01.645010 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.644978 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.646232 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.646213 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"] Nov 02 13:43:01.646314 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.646288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.646916 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.646897 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 13:43:01.646916 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.646912 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 13:43:01.647054 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.646919 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 13:43:01.647054 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.646924 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6lw86\"" Nov 02 13:43:01.648715 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.648697 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 13:43:01.681725 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.681702 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 13:43:01.681818 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.681763 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d8sw9" Nov 02 13:43:01.692900 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.692883 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 13:43:01.696721 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.696696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.696818 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.696771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5291323c-ea92-499c-b246-a4f692f3f8c5-host\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.696863 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.696833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:01.696919 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.696898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b29d373a-4cfd-4d5b-8561-e604264326db-iptables-alerter-script\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.696965 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.696936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovn-node-metrics-cert\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.697027 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.696961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-system-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.697106 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.697160 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-multus-certs\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.697212 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkkz\" (UniqueName: \"kubernetes.io/projected/16e35961-5e37-4e7b-bf8d-5a8d7c273893-kube-api-access-drkkz\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.697268 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4xv\" (UniqueName: \"kubernetes.io/projected/b29d373a-4cfd-4d5b-8561-e604264326db-kube-api-access-ml4xv\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.697314 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8x5\" (UniqueName: \"kubernetes.io/projected/36315738-ca55-4bbb-8644-748e4ae00efe-kube-api-access-zq8x5\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.697350 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-system-cni-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.697388 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.697466 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-systemd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.697547 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-config\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.697547 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-env-overrides\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.697738 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-lib-modules\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.697738 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:01.697738 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-kubelet-config\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.697738 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-sys-fs\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.697738 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-etc-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.697920 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.697920 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-conf\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.697920 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-systemd\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.697920 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697895 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-hostroot\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.698119 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.697927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/790cdff4-1780-4548-a327-e7b7df474347-hosts-file\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.698119 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.698119 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-kubelet\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.698119 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-systemd-units\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.698119 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-slash\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.698337 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-netns\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.698337 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-log-socket\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.698337 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-sys\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.698337 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.698337 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-registration-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698340 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d8sw9" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb826\" (UniqueName: \"kubernetes.io/projected/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kube-api-access-jb826\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b29d373a-4cfd-4d5b-8561-e604264326db-host-slash\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-script-lib\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-kubernetes\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/5291323c-ea92-499c-b246-a4f692f3f8c5-kube-api-access-8x5g6\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-bin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.698640 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-bin\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-etc-tuned\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-cni-binary-copy\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-conf-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790cdff4-1780-4548-a327-e7b7df474347-tmp-dir\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-socket-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-host\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b34d67be-03c6-42d2-8e6f-d5936d7d940b-konnectivity-ca\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:01.698959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-dbus\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-ovn\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.698993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-node-log\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysconfig\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-var-lib-kubelet\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-device-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-var-lib-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-netd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-cnibin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.699322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-kubelet\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jtm\" (UniqueName: \"kubernetes.io/projected/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-kube-api-access-79jtm\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-multus\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsdw\" (UniqueName: \"kubernetes.io/projected/88b783e5-08d6-44c6-ac37-b50adf8736ea-kube-api-access-8wsdw\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cnibin\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xt65\" (UniqueName: \"kubernetes.io/projected/790cdff4-1780-4548-a327-e7b7df474347-kube-api-access-4xt65\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-modprobe-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-run\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-socket-dir-parent\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.699726 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-k8s-cni-cncf-io\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-netns\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-etc-kubernetes\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699798 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-tmp\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5291323c-ea92-499c-b246-a4f692f3f8c5-serviceca\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b34d67be-03c6-42d2-8e6f-d5936d7d940b-agent-certs\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.699956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-daemon-config\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.700045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.700004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkp9k\" (UniqueName: \"kubernetes.io/projected/814f4474-5a86-4ddf-8bd8-beeebe917d71-kube-api-access-fkp9k\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.705007 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:01.704968 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bef5910d750df47fec5174fe909f02.slice/crio-1d7ca799ea3892c4ea1b4495d403cc484b6e5f5d05d7f529f286b04698d77783 WatchSource:0}: Error finding container 1d7ca799ea3892c4ea1b4495d403cc484b6e5f5d05d7f529f286b04698d77783: Status 404 returned error can't find the container with id 1d7ca799ea3892c4ea1b4495d403cc484b6e5f5d05d7f529f286b04698d77783 Nov 02 13:43:01.705391 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:01.705371 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82d0dbde7394ec25631244e1f1d5cd9.slice/crio-71fbdaae8e4207f512c28ea54b6db30c72bc3a0c775210fba12a1f0586e655ba WatchSource:0}: Error finding container 71fbdaae8e4207f512c28ea54b6db30c72bc3a0c775210fba12a1f0586e655ba: Status 404 returned error can't find the container with id 71fbdaae8e4207f512c28ea54b6db30c72bc3a0c775210fba12a1f0586e655ba Nov 02 13:43:01.711486 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.711469 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 13:43:01.719914 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.719894 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"] Nov 02 13:43:01.723288 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.720777 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:43:01.723288 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.722021 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" event={"ID":"e82d0dbde7394ec25631244e1f1d5cd9","Type":"ContainerStarted","Data":"71fbdaae8e4207f512c28ea54b6db30c72bc3a0c775210fba12a1f0586e655ba"} Nov 02 13:43:01.724646 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.724613 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerStarted","Data":"1d7ca799ea3892c4ea1b4495d403cc484b6e5f5d05d7f529f286b04698d77783"} Nov 02 13:43:01.757745 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.757711 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 13:43:01.757745 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.757744 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 13:43:01.757981 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.757902 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 13:43:01.757981 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.757947 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:43:01.757981 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.757968 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 13:43:01.758162 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.758007 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 13:43:01.758162 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.758032 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 13:43:01.758162 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.758055 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 13:43:01.758162 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.758080 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 13:43:01.758162 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.758148 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 13:43:01.758441 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.758428 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 13:43:01.758554 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.758544 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 13:43:01.759071 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759056 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 13:43:01.759071 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759067 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 13:43:01.759208 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759191 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:43:01.759247 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759198 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 13:43:01.759247 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759223 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 13:43:01.759390 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759378 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 13:43:01.759489 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759475 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 13:43:01.759531 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759498 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 13:43:01.759531 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759516 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 13:43:01.759658 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759644 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-52vg6\"" Nov 02 13:43:01.759705 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759692 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-47zgd\"" Nov 02 13:43:01.759759 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759745 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bqvhh\"" Nov 02 13:43:01.759793 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759776 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 13:43:01.759841 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759825 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5dqzs\"" Nov 02 13:43:01.759880 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759848 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qnz8k\"" Nov 02 13:43:01.759910 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759887 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w4l5x\"" Nov 02 13:43:01.760009 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.759996 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lvdl4\"" Nov 02 13:43:01.760067 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.760033 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4wcws\"" Nov 02 13:43:01.770677 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.770658 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 13:43:01.800550 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-dbus\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.800643 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.800643 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-ovn\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.800643 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-node-log\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.800751 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysconfig\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.800751 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-var-lib-kubelet\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.800820 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800746 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-dbus\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.800820 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-device-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.800820 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-var-lib-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.800820 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800783 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-node-log\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.800820 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-netd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.800820 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800809 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysconfig\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-device-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800812 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-ovn\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800853 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-var-lib-kubelet\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800857 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-netd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800863 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-var-lib-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.800917 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-cnibin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-kubelet\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.800967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.800997 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret podName:a14e8823-8f98-40c4-b20d-dfea320c8304 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:02.300968134 +0000 UTC m=+2.096403538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret") pod "global-pull-secret-syncer-scx6x" (UID: "a14e8823-8f98-40c4-b20d-dfea320c8304") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:01.801043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801036 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-kubelet\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jtm\" (UniqueName: \"kubernetes.io/projected/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-kube-api-access-79jtm\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-multus\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsdw\" (UniqueName: \"kubernetes.io/projected/88b783e5-08d6-44c6-ac37-b50adf8736ea-kube-api-access-8wsdw\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cnibin\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801139 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-cnibin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xt65\" (UniqueName: \"kubernetes.io/projected/790cdff4-1780-4548-a327-e7b7df474347-kube-api-access-4xt65\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-multus\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cnibin\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-modprobe-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-run\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-socket-dir-parent\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-k8s-cni-cncf-io\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-netns\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801481 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-run\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-k8s-cni-cncf-io\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-etc-kubernetes\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801503 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-socket-dir-parent\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.801521 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801513 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-modprobe-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801528 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-netns\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801557 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801557 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801568 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-etc-kubernetes\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-tmp\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5291323c-ea92-499c-b246-a4f692f3f8c5-serviceca\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b34d67be-03c6-42d2-8e6f-d5936d7d940b-agent-certs\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-daemon-config\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkp9k\" (UniqueName: \"kubernetes.io/projected/814f4474-5a86-4ddf-8bd8-beeebe917d71-kube-api-access-fkp9k\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5291323c-ea92-499c-b246-a4f692f3f8c5-host\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b29d373a-4cfd-4d5b-8561-e604264326db-iptables-alerter-script\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801871 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovn-node-metrics-cert\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.802402 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801936 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.801924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-system-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-system-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-multus-certs\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkkz\" (UniqueName: \"kubernetes.io/projected/16e35961-5e37-4e7b-bf8d-5a8d7c273893-kube-api-access-drkkz\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4xv\" (UniqueName: \"kubernetes.io/projected/b29d373a-4cfd-4d5b-8561-e604264326db-kube-api-access-ml4xv\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8x5\" (UniqueName: \"kubernetes.io/projected/36315738-ca55-4bbb-8644-748e4ae00efe-kube-api-access-zq8x5\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-system-cni-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802231 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5291323c-ea92-499c-b246-a4f692f3f8c5-serviceca\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-systemd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-config\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-env-overrides\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802332 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5291323c-ea92-499c-b246-a4f692f3f8c5-host\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-lib-modules\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.804802 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-kubelet-config\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-sys-fs\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-etc-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802484 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-lib-modules\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802537 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-multus-certs\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-system-cni-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.802611 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802654 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-etc-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.802676 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs podName:88b783e5-08d6-44c6-ac37-b50adf8736ea nodeName:}" failed. No retries permitted until 2026-11-02 13:43:02.302659975 +0000 UTC m=+2.098095396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs") pod "network-metrics-daemon-v8n7n" (UID: "88b783e5-08d6-44c6-ac37-b50adf8736ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-systemd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802731 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-sys-fs\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-conf\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-systemd\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802790 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.805712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-hostroot\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802829 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-hostroot\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/790cdff4-1780-4548-a327-e7b7df474347-hosts-file\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-kubelet-config\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-kubelet\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802912 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/790cdff4-1780-4548-a327-e7b7df474347-hosts-file\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-systemd-units\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-slash\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.802983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-kubelet\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803002 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-systemd\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-netns\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803070 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-systemd-units\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-log-socket\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-sys\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803118 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-conf\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.806915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803135 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-log-socket\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803105 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-netns\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-registration-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb826\" (UniqueName: \"kubernetes.io/projected/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kube-api-access-jb826\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b29d373a-4cfd-4d5b-8561-e604264326db-host-slash\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803232 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b29d373a-4cfd-4d5b-8561-e604264326db-iptables-alerter-script\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-script-lib\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803240 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-registration-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803186 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-sys\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-kubernetes\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803292 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803315 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-kubernetes\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/5291323c-ea92-499c-b246-a4f692f3f8c5-kube-api-access-8x5g6\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803359 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b29d373a-4cfd-4d5b-8561-e604264326db-host-slash\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-bin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-bin\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807476 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803403 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-slash\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-etc-tuned\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-cni-binary-copy\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-conf-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790cdff4-1780-4548-a327-e7b7df474347-tmp-dir\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803537 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-bin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-socket-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-host\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b34d67be-03c6-42d2-8e6f-d5936d7d940b-konnectivity-ca\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-env-overrides\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.803987 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-conf-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804014 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-script-lib\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804051 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-bin\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804049 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-config\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.807959 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804202 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-socket-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804306 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b34d67be-03c6-42d2-8e6f-d5936d7d940b-konnectivity-ca\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804378 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804439 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-host\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804514 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790cdff4-1780-4548-a327-e7b7df474347-tmp-dir\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804514 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-daemon-config\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.804884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-cni-binary-copy\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.806249 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-tmp\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.806329 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovn-node-metrics-cert\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.806420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b34d67be-03c6-42d2-8e6f-d5936d7d940b-agent-certs\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:01.808460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.806982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-etc-tuned\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.826275 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.826247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xt65\" (UniqueName: \"kubernetes.io/projected/790cdff4-1780-4548-a327-e7b7df474347-kube-api-access-4xt65\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:01.826542 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.826519 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsdw\" (UniqueName: \"kubernetes.io/projected/88b783e5-08d6-44c6-ac37-b50adf8736ea-kube-api-access-8wsdw\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:01.826784 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.826754 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:43:01.826784 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.826775 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:43:01.826945 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.826790 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wvzgw for pod openshift-network-diagnostics/network-check-target-cc5c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:01.826945 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:01.826841 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw podName:9cb24650-63d2-4d50-9a20-db9bf4a5e2c9 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:02.32682604 +0000 UTC m=+2.122261455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wvzgw" (UniqueName: "kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw") pod "network-check-target-cc5c8" (UID: "9cb24650-63d2-4d50-9a20-db9bf4a5e2c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:01.826945 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.826935 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jtm\" (UniqueName: \"kubernetes.io/projected/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-kube-api-access-79jtm\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:01.829796 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.829768 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb826\" (UniqueName: \"kubernetes.io/projected/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kube-api-access-jb826\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:01.830220 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.830191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkkz\" (UniqueName: \"kubernetes.io/projected/16e35961-5e37-4e7b-bf8d-5a8d7c273893-kube-api-access-drkkz\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.830291 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.830196 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8x5\" (UniqueName: \"kubernetes.io/projected/36315738-ca55-4bbb-8644-748e4ae00efe-kube-api-access-zq8x5\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:01.830539 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.830524 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkp9k\" (UniqueName: \"kubernetes.io/projected/814f4474-5a86-4ddf-8bd8-beeebe917d71-kube-api-access-fkp9k\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:43:01.830583 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.830542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4xv\" (UniqueName: \"kubernetes.io/projected/b29d373a-4cfd-4d5b-8561-e604264326db-kube-api-access-ml4xv\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:01.830823 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.830805 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/5291323c-ea92-499c-b246-a4f692f3f8c5-kube-api-access-8x5g6\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.952739 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.952617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:43:01.961621 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:01.961589 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5291323c_ea92_499c_b246_a4f692f3f8c5.slice/crio-0e0d4d8a8a7c4fa08c9e19f47c0a644c400e19336a7a83c887d29deeebebaf4d WatchSource:0}: Error finding container 0e0d4d8a8a7c4fa08c9e19f47c0a644c400e19336a7a83c887d29deeebebaf4d: Status 404 returned error can't find the container with id 0e0d4d8a8a7c4fa08c9e19f47c0a644c400e19336a7a83c887d29deeebebaf4d Nov 02 13:43:01.978271 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.978246 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:43:01.979982 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:01.979950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:01.988122 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:01.988093 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e35961_5e37_4e7b_bf8d_5a8d7c273893.slice/crio-12ba0dc43d43f4899ac81c7c60b103ec4137895ff6c963087c531e717d539d56 WatchSource:0}: Error finding container 12ba0dc43d43f4899ac81c7c60b103ec4137895ff6c963087c531e717d539d56: Status 404 returned error can't find the container with id 12ba0dc43d43f4899ac81c7c60b103ec4137895ff6c963087c531e717d539d56 Nov 02 13:43:01.988794 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:01.988767 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34d67be_03c6_42d2_8e6f_d5936d7d940b.slice/crio-80565425f70ad633a65bab7ce1d5af11122c5830bb78de1737297d3b4a7a8dcd WatchSource:0}: Error finding container 80565425f70ad633a65bab7ce1d5af11122c5830bb78de1737297d3b4a7a8dcd: Status 404 returned error can't find the container with id 80565425f70ad633a65bab7ce1d5af11122c5830bb78de1737297d3b4a7a8dcd Nov 02 13:43:02.000386 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.000365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:43:02.008574 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:02.008552 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fad09e_a1b4_4998_bb14_8e6e33a3a03c.slice/crio-6caba99e97540d491261dd7a170d10ff675ac0a6ccc14a1feef24fb6e9600bdb WatchSource:0}: Error finding container 6caba99e97540d491261dd7a170d10ff675ac0a6ccc14a1feef24fb6e9600bdb: Status 404 returned error can't find the container with id 6caba99e97540d491261dd7a170d10ff675ac0a6ccc14a1feef24fb6e9600bdb Nov 02 13:43:02.015225 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.015206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:43:02.022973 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:02.022950 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29d373a_4cfd_4d5b_8561_e604264326db.slice/crio-ccb8ebaf7df6a5841a53771b5ebfd0caf034c96f71b441bb9d37a1605e28906c WatchSource:0}: Error finding container ccb8ebaf7df6a5841a53771b5ebfd0caf034c96f71b441bb9d37a1605e28906c: Status 404 returned error can't find the container with id ccb8ebaf7df6a5841a53771b5ebfd0caf034c96f71b441bb9d37a1605e28906c Nov 02 13:43:02.031278 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.031251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:02.032350 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.032303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g268n" Nov 02 13:43:02.038329 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.038305 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:43:02.042130 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.042104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:43:02.043090 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:02.043063 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52eda2b9_8258_4f7e_bc72_6ad9f6f1167a.slice/crio-b7688829976fff75a5c9900b8ae22fb29699368757ec7d6cba1610fb787f5f03 WatchSource:0}: Error finding container b7688829976fff75a5c9900b8ae22fb29699368757ec7d6cba1610fb787f5f03: Status 404 returned error can't find the container with id b7688829976fff75a5c9900b8ae22fb29699368757ec7d6cba1610fb787f5f03 Nov 02 13:43:02.049853 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:02.049819 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814f4474_5a86_4ddf_8bd8_beeebe917d71.slice/crio-e2e469850889f5454ef32b7c2a7741f5a31573051185b3e7a5e8778b2874f7f4 WatchSource:0}: Error finding container e2e469850889f5454ef32b7c2a7741f5a31573051185b3e7a5e8778b2874f7f4: Status 404 returned error can't find the container with id e2e469850889f5454ef32b7c2a7741f5a31573051185b3e7a5e8778b2874f7f4 Nov 02 13:43:02.053209 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:02.051350 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36315738_ca55_4bbb_8644_748e4ae00efe.slice/crio-e5d571ccdaa0945dc1544c0a8fdcdc515abad6eb6261c6c951725e13d7ac2565 WatchSource:0}: Error finding container e5d571ccdaa0945dc1544c0a8fdcdc515abad6eb6261c6c951725e13d7ac2565: Status 404 returned error can't find the container with id e5d571ccdaa0945dc1544c0a8fdcdc515abad6eb6261c6c951725e13d7ac2565 Nov 02 13:43:02.056067 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:02.056031 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790cdff4_1780_4548_a327_e7b7df474347.slice/crio-2c484b97bf8b5f0634d95075477c05abe6a73beb822241db51b0421b2152e197 WatchSource:0}: Error finding container 2c484b97bf8b5f0634d95075477c05abe6a73beb822241db51b0421b2152e197: Status 404 returned error can't find the container with id 2c484b97bf8b5f0634d95075477c05abe6a73beb822241db51b0421b2152e197 Nov 02 13:43:02.308605 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.308518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:02.308791 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.308614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:02.308791 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:02.308739 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:02.308889 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:02.308800 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret podName:a14e8823-8f98-40c4-b20d-dfea320c8304 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:03.308782179 +0000 UTC m=+3.104217595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret") pod "global-pull-secret-syncer-scx6x" (UID: "a14e8823-8f98-40c4-b20d-dfea320c8304") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:02.309224 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:02.309206 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:02.309309 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:02.309294 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs podName:88b783e5-08d6-44c6-ac37-b50adf8736ea nodeName:}" failed. No retries permitted until 2026-11-02 13:43:03.309250124 +0000 UTC m=+3.104685549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs") pod "network-metrics-daemon-v8n7n" (UID: "88b783e5-08d6-44c6-ac37-b50adf8736ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:02.341301 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.341270 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:43:02.409745 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.409684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:02.409935 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:02.409910 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:43:02.410043 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:02.409939 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:43:02.410043 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:02.409952 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wvzgw for pod openshift-network-diagnostics/network-check-target-cc5c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:02.410043 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:02.410029 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw podName:9cb24650-63d2-4d50-9a20-db9bf4a5e2c9 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:03.410011242 +0000 UTC m=+3.205446647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvzgw" (UniqueName: "kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw") pod "network-check-target-cc5c8" (UID: "9cb24650-63d2-4d50-9a20-db9bf4a5e2c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:02.700559 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.700341 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:38:01 +0000 UTC" deadline="2028-04-19 15:52:22.605260852 +0000 UTC" Nov 02 13:43:02.700559 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.700458 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12818h9m19.90480779s" Nov 02 13:43:02.742200 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.742167 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8hhsr" event={"ID":"b34d67be-03c6-42d2-8e6f-d5936d7d940b","Type":"ContainerStarted","Data":"80565425f70ad633a65bab7ce1d5af11122c5830bb78de1737297d3b4a7a8dcd"} Nov 02 13:43:02.752625 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.752581 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerStarted","Data":"12ba0dc43d43f4899ac81c7c60b103ec4137895ff6c963087c531e717d539d56"} Nov 02 13:43:02.778786 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.778674 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"b7688829976fff75a5c9900b8ae22fb29699368757ec7d6cba1610fb787f5f03"} Nov 02 13:43:02.806262 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.806219 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ff8sv" event={"ID":"b29d373a-4cfd-4d5b-8561-e604264326db","Type":"ContainerStarted","Data":"ccb8ebaf7df6a5841a53771b5ebfd0caf034c96f71b441bb9d37a1605e28906c"} Nov 02 13:43:02.820375 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.820287 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9d7n9" event={"ID":"5291323c-ea92-499c-b246-a4f692f3f8c5","Type":"ContainerStarted","Data":"0e0d4d8a8a7c4fa08c9e19f47c0a644c400e19336a7a83c887d29deeebebaf4d"} Nov 02 13:43:02.829422 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.829367 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9lm6" event={"ID":"790cdff4-1780-4548-a327-e7b7df474347","Type":"ContainerStarted","Data":"2c484b97bf8b5f0634d95075477c05abe6a73beb822241db51b0421b2152e197"} Nov 02 13:43:02.845080 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.840154 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" event={"ID":"36315738-ca55-4bbb-8644-748e4ae00efe","Type":"ContainerStarted","Data":"e5d571ccdaa0945dc1544c0a8fdcdc515abad6eb6261c6c951725e13d7ac2565"} Nov 02 13:43:02.853085 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.853043 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g268n" event={"ID":"814f4474-5a86-4ddf-8bd8-beeebe917d71","Type":"ContainerStarted","Data":"e2e469850889f5454ef32b7c2a7741f5a31573051185b3e7a5e8778b2874f7f4"} Nov 02 13:43:02.857714 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.857683 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" event={"ID":"33fad09e-a1b4-4998-bb14-8e6e33a3a03c","Type":"ContainerStarted","Data":"6caba99e97540d491261dd7a170d10ff675ac0a6ccc14a1feef24fb6e9600bdb"} Nov 02 13:43:02.993523 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:02.993445 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:43:03.318416 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.318332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:03.318416 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.318403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:03.318628 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.318566 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:03.318714 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.318630 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs podName:88b783e5-08d6-44c6-ac37-b50adf8736ea nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.31861139 +0000 UTC m=+5.114046804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs") pod "network-metrics-daemon-v8n7n" (UID: "88b783e5-08d6-44c6-ac37-b50adf8736ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:03.319061 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.319043 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:03.319139 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.319097 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret podName:a14e8823-8f98-40c4-b20d-dfea320c8304 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.319080662 +0000 UTC m=+5.114516072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret") pod "global-pull-secret-syncer-scx6x" (UID: "a14e8823-8f98-40c4-b20d-dfea320c8304") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:03.419764 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.419717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:03.419942 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.419924 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:43:03.420049 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.419950 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:43:03.420049 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.419963 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wvzgw for pod openshift-network-diagnostics/network-check-target-cc5c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:03.420049 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.420034 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw podName:9cb24650-63d2-4d50-9a20-db9bf4a5e2c9 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.42001596 +0000 UTC m=+5.215451374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvzgw" (UniqueName: "kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw") pod "network-check-target-cc5c8" (UID: "9cb24650-63d2-4d50-9a20-db9bf4a5e2c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:03.701757 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.701644 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:38:01 +0000 UTC" deadline="2028-04-17 05:15:42.790422478 +0000 UTC" Nov 02 13:43:03.701757 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.701716 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12759h32m39.088711211s" Nov 02 13:43:03.718368 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.718245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:03.718502 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.718379 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:03.718729 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.718712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:03.718830 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.718785 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:03.718830 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.718830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:03.718925 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:03.718871 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:03.863466 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:03.863431 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:43:05.335339 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.335306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:05.335775 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.335373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:05.335775 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.335482 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:05.335775 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.335525 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs podName:88b783e5-08d6-44c6-ac37-b50adf8736ea nodeName:}" failed. No retries permitted until 2026-11-02 13:43:09.335512054 +0000 UTC m=+9.130947454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs") pod "network-metrics-daemon-v8n7n" (UID: "88b783e5-08d6-44c6-ac37-b50adf8736ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:05.335883 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.335795 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:05.335883 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.335826 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret podName:a14e8823-8f98-40c4-b20d-dfea320c8304 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:09.335815933 +0000 UTC m=+9.131251332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret") pod "global-pull-secret-syncer-scx6x" (UID: "a14e8823-8f98-40c4-b20d-dfea320c8304") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:05.436735 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.436502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:05.436925 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.436905 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:43:05.437003 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.436932 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:43:05.437003 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.436951 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wvzgw for pod openshift-network-diagnostics/network-check-target-cc5c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:05.437166 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.437029 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw podName:9cb24650-63d2-4d50-9a20-db9bf4a5e2c9 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:09.437009838 +0000 UTC m=+9.232445246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvzgw" (UniqueName: "kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw") pod "network-check-target-cc5c8" (UID: "9cb24650-63d2-4d50-9a20-db9bf4a5e2c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:05.718827 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.718801 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:05.718965 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.718937 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:05.719048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.719014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:05.719107 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.719082 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:05.719259 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.719121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:05.719259 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:05.719183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:05.868928 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.868891 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerStarted","Data":"677f5e07c18fd4f100458255948a8ae11262536d272f7e22d5007ef505871f29"} Nov 02 13:43:05.871929 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.871880 2579 generic.go:356] "Generic (PLEG): container finished" podID="60bef5910d750df47fec5174fe909f02" containerID="48fff5c9b870ae9ae4f525158b5653b2b3a4bd9a27eae95dae9a14941aec6e9e" exitCode=0 Nov 02 13:43:05.872059 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.871958 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerDied","Data":"48fff5c9b870ae9ae4f525158b5653b2b3a4bd9a27eae95dae9a14941aec6e9e"} Nov 02 13:43:05.877792 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:05.877441 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" event={"ID":"e82d0dbde7394ec25631244e1f1d5cd9","Type":"ContainerStarted","Data":"10b90b2a1003f3a3f7d1a8cf25e4c1071f7239147f0113051e31f42e73405120"} Nov 02 13:43:06.883968 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:06.883874 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8hhsr" event={"ID":"b34d67be-03c6-42d2-8e6f-d5936d7d940b","Type":"ContainerStarted","Data":"274af47a04fe1c826dc6280f38836dd10d27e81d4e7466e26d12ab95620c6a76"} Nov 02 13:43:06.885772 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:06.885734 2579 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="677f5e07c18fd4f100458255948a8ae11262536d272f7e22d5007ef505871f29" exitCode=0 Nov 02 13:43:06.885888 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:06.885811 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerDied","Data":"677f5e07c18fd4f100458255948a8ae11262536d272f7e22d5007ef505871f29"} Nov 02 13:43:06.891869 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:06.891827 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerStarted","Data":"f1f7146ba8df40a865cc2ab7e0239327d6d65ab90acb88a903999aee9344cf81"} Nov 02 13:43:06.924793 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:06.924734 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" podStartSLOduration=5.924716338 podStartE2EDuration="5.924716338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:06.020118941 +0000 UTC m=+5.815554359" watchObservedRunningTime="2026-11-02 13:43:06.924716338 +0000 UTC m=+6.720151763" Nov 02 13:43:06.964853 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:06.964782 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8hhsr" podStartSLOduration=2.506782202 podStartE2EDuration="6.964759227s" totalImagesPullingTime="4.457977025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:06.925066012 +0000 UTC m=+6.720501447" watchObservedRunningTime="2026-11-02 13:43:06.964759227 +0000 UTC m=+6.760194648" Nov 02 13:43:06.965872 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:06.965830 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" podStartSLOduration=5.965812931 podStartE2EDuration="5.965812931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:06.964507264 +0000 UTC m=+6.759942687" watchObservedRunningTime="2026-11-02 13:43:06.965812931 +0000 UTC m=+6.761248355" Nov 02 13:43:07.439848 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:07.439772 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:07.440840 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:07.440573 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:07.719996 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:07.719126 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:07.719996 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:07.719255 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:07.719996 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:07.719656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:07.719996 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:07.719758 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:07.719996 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:07.719824 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:07.719996 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:07.719910 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:07.899528 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:07.899037 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9d7n9" event={"ID":"5291323c-ea92-499c-b246-a4f692f3f8c5","Type":"ContainerStarted","Data":"87158138669ee42c384f4d0f3ac6856ffa90f48b041c695cd539984354fc5208"} Nov 02 13:43:07.938448 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:07.938012 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9d7n9" podStartSLOduration=2.836373084 podStartE2EDuration="7.937966868s" totalImagesPullingTime="5.101593784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:07.937181488 +0000 UTC m=+7.732616914" watchObservedRunningTime="2026-11-02 13:43:07.937966868 +0000 UTC m=+7.733402290" Nov 02 13:43:08.908562 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:08.908082 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" event={"ID":"33fad09e-a1b4-4998-bb14-8e6e33a3a03c","Type":"ContainerStarted","Data":"767908fe3bc8c2850ba30c17518c44898272435ebb54189a3b0213329e5bbf1b"} Nov 02 13:43:08.908562 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:08.908191 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:43:09.374041 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.373241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:09.374041 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.373328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:09.374041 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.373489 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:09.374041 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.373563 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs podName:88b783e5-08d6-44c6-ac37-b50adf8736ea nodeName:}" failed. No retries permitted until 2026-11-02 13:43:17.373542851 +0000 UTC m=+17.168978251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs") pod "network-metrics-daemon-v8n7n" (UID: "88b783e5-08d6-44c6-ac37-b50adf8736ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:09.374041 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.373950 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:09.374041 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.374011 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret podName:a14e8823-8f98-40c4-b20d-dfea320c8304 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:17.373995987 +0000 UTC m=+17.169431413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret") pod "global-pull-secret-syncer-scx6x" (UID: "a14e8823-8f98-40c4-b20d-dfea320c8304") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:09.474349 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.473694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:09.474349 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.473868 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:43:09.474349 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.473888 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:43:09.474349 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.473902 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wvzgw for pod openshift-network-diagnostics/network-check-target-cc5c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:09.474349 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.473963 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw podName:9cb24650-63d2-4d50-9a20-db9bf4a5e2c9 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:17.473945065 +0000 UTC m=+17.269380482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvzgw" (UniqueName: "kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw") pod "network-check-target-cc5c8" (UID: "9cb24650-63d2-4d50-9a20-db9bf4a5e2c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:09.719020 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.718764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:09.719020 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.718786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:09.719020 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.718809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:09.719020 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.718921 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:09.719292 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.719054 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:09.719292 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:09.719157 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:09.867626 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.867527 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 13:43:09.912607 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.911932 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g268n" event={"ID":"814f4474-5a86-4ddf-8bd8-beeebe917d71","Type":"ContainerStarted","Data":"4644ab3fdd62b04e06dde20880b2a636329181ccd945bc2536e656f9afeaca9e"} Nov 02 13:43:09.916442 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.916067 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" event={"ID":"33fad09e-a1b4-4998-bb14-8e6e33a3a03c","Type":"ContainerStarted","Data":"3325cb65ce85b9b4aaa1616ce8b9eff3239295015ba1e314e0e45d038e924a69"} Nov 02 13:43:09.956537 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:09.956475 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-g268n" podStartSLOduration=2.221260115 podStartE2EDuration="9.956461907s" totalImagesPullingTime="7.735201792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:09.95589507 +0000 UTC m=+9.751330491" watchObservedRunningTime="2026-11-02 13:43:09.956461907 +0000 UTC m=+9.751897327" Nov 02 13:43:10.663666 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:10.663551 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:43:09.8675432Z","UUID":"f416cb1a-6d30-46b9-8586-693f3ac8a9df","Handler":null,"Name":"","Endpoint":""} Nov 02 13:43:10.667328 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:10.667303 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 13:43:10.667460 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:10.667339 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 13:43:10.923625 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:10.922068 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerStarted","Data":"2079f19251417a9a2e283191b94da176f0fd5bdf980ab5448eac9ddab363689a"} Nov 02 13:43:10.933012 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:10.926558 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9lm6" event={"ID":"790cdff4-1780-4548-a327-e7b7df474347","Type":"ContainerStarted","Data":"0a5ef88b1f1a62ef8cdb2ee382482b1bf1ec8174b8c4535a3622422e7fabee41"} Nov 02 13:43:11.019720 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:11.019669 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-j9lm6" podStartSLOduration=1.917086661 podStartE2EDuration="10.019654644s" totalImagesPullingTime="8.102567983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:11.019583944 +0000 UTC m=+10.815019365" watchObservedRunningTime="2026-11-02 13:43:11.019654644 +0000 UTC m=+10.815090065" Nov 02 13:43:11.718663 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:11.718628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:11.718845 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:11.718749 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:11.719147 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:11.719127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:11.719250 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:11.719232 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:11.719322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:11.719296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:11.719403 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:11.719383 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:11.931914 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:11.931176 2579 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="2079f19251417a9a2e283191b94da176f0fd5bdf980ab5448eac9ddab363689a" exitCode=0 Nov 02 13:43:11.931914 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:11.931284 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerDied","Data":"2079f19251417a9a2e283191b94da176f0fd5bdf980ab5448eac9ddab363689a"} Nov 02 13:43:11.934542 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:11.934447 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ff8sv" event={"ID":"b29d373a-4cfd-4d5b-8561-e604264326db","Type":"ContainerStarted","Data":"e215fc2645c45a70bc415e6f0bbb8e55181a63888824a03fec896148d17184f3"} Nov 02 13:43:11.937343 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:11.937319 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" event={"ID":"33fad09e-a1b4-4998-bb14-8e6e33a3a03c","Type":"ContainerStarted","Data":"1d25ad69bf1c600fdb9458439bdb1fe8b073ee33129ede7ac7e0ea18feb499c8"} Nov 02 13:43:12.020675 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:12.020621 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" podStartSLOduration=3.538084572 podStartE2EDuration="12.020605272s" totalImagesPullingTime="8.4825207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:12.02047223 +0000 UTC m=+11.815907651" watchObservedRunningTime="2026-11-02 13:43:12.020605272 +0000 UTC m=+11.816040697" Nov 02 13:43:12.059309 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:12.059259 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ff8sv" podStartSLOduration=3.848523161 podStartE2EDuration="12.05924312s" totalImagesPullingTime="8.210719959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:12.059229698 +0000 UTC m=+11.854665123" watchObservedRunningTime="2026-11-02 13:43:12.05924312 +0000 UTC m=+11.854678541" Nov 02 13:43:12.941442 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:12.941359 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" event={"ID":"36315738-ca55-4bbb-8644-748e4ae00efe","Type":"ContainerStarted","Data":"549e22e2b3aa1152c940407f560e7afd8472fd850eff10a3cc935f45fd614fea"} Nov 02 13:43:12.944176 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:12.944138 2579 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="555e0dd5e4151b7404d4d76f5c8fd4e41aa6bc56e2fc2b108758e9b11f8bcf1b" exitCode=0 Nov 02 13:43:12.944297 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:12.944209 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerDied","Data":"555e0dd5e4151b7404d4d76f5c8fd4e41aa6bc56e2fc2b108758e9b11f8bcf1b"} Nov 02 13:43:12.983895 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:12.983850 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" podStartSLOduration=1.710443916 podStartE2EDuration="11.98383696s" totalImagesPullingTime="10.273393044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:12.983504258 +0000 UTC m=+12.778939680" watchObservedRunningTime="2026-11-02 13:43:12.98383696 +0000 UTC m=+12.779272381" Nov 02 13:43:13.718591 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:13.718556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:13.718756 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:13.718594 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:13.718756 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:13.718671 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:13.718756 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:13.718735 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:13.718911 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:13.718866 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:13.718976 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:13.718956 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:14.951704 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:14.951667 2579 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="e6c446d927ceaf665d1f089efa86c38c99a5d9e1993a23001c7c5a37474e09ab" exitCode=0 Nov 02 13:43:14.952377 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:14.951716 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerDied","Data":"e6c446d927ceaf665d1f089efa86c38c99a5d9e1993a23001c7c5a37474e09ab"} Nov 02 13:43:15.718322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:15.718282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:15.718322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:15.718304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:15.718549 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:15.718434 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:15.718549 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:15.718480 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:15.718804 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:15.718773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:15.718880 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:15.718859 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:16.958009 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:16.957794 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"931aabd1284eaf8afe8757f7e790a969d94ba44287b853c84591eac24e94e7fe"} Nov 02 13:43:16.961167 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:16.961143 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerStarted","Data":"de5c5740c87a5894570a178971801fccccd1ff4a42593758c3b5aab3ab28a523"} Nov 02 13:43:17.442329 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.442117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:17.442329 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.442269 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:17.442329 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.442311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:17.442329 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.442332 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret podName:a14e8823-8f98-40c4-b20d-dfea320c8304 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:33.442317424 +0000 UTC m=+33.237752844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret") pod "global-pull-secret-syncer-scx6x" (UID: "a14e8823-8f98-40c4-b20d-dfea320c8304") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:17.442563 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.442378 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:17.442563 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.442417 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs podName:88b783e5-08d6-44c6-ac37-b50adf8736ea nodeName:}" failed. No retries permitted until 2026-11-02 13:43:33.442406903 +0000 UTC m=+33.237842303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs") pod "network-metrics-daemon-v8n7n" (UID: "88b783e5-08d6-44c6-ac37-b50adf8736ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:17.542820 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.542790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:17.543018 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.542956 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:43:17.543018 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.542977 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:43:17.543018 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.543001 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wvzgw for pod openshift-network-diagnostics/network-check-target-cc5c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:17.543114 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.543048 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw podName:9cb24650-63d2-4d50-9a20-db9bf4a5e2c9 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:33.543035808 +0000 UTC m=+33.338471226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvzgw" (UniqueName: "kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw") pod "network-check-target-cc5c8" (UID: "9cb24650-63d2-4d50-9a20-db9bf4a5e2c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:17.718622 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.718544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:17.718786 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.718542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:17.718786 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.718650 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:17.718786 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.718542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:17.718786 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.718734 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:17.718938 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:17.718811 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:17.965794 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.965767 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:43:17.966187 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.966129 2579 generic.go:356] "Generic (PLEG): container finished" podID="52eda2b9-8258-4f7e-bc72-6ad9f6f1167a" containerID="d7a60ba7d2f5a5d78d0d248c98a0eeb8325a9f1105b2b849c6e46dc0d44fd514" exitCode=1 Nov 02 13:43:17.966253 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.966201 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"6c1491fe2c6f8c6b8f019c19783f25647671f6438d62d8db3cb83ae0c7ce4795"} Nov 02 13:43:17.966315 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.966299 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"b5b1674b14eed32b4c85c7371ece97301a93515129d959d757247ee22d200fa5"} Nov 02 13:43:17.966368 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.966338 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"d49d99b40d447ae2829758c3705599be1a843bbf2f012c7773269089c6a97197"} Nov 02 13:43:17.966368 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.966366 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"c1a54b4bdeea9bb7e4b755decfc2259a97c4d5eb7ca87bf9b2eb466ba0dbd083"} Nov 02 13:43:17.966444 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.966394 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerDied","Data":"d7a60ba7d2f5a5d78d0d248c98a0eeb8325a9f1105b2b849c6e46dc0d44fd514"} Nov 02 13:43:17.968879 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.968822 2579 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="de5c5740c87a5894570a178971801fccccd1ff4a42593758c3b5aab3ab28a523" exitCode=0 Nov 02 13:43:17.968879 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:17.968854 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerDied","Data":"de5c5740c87a5894570a178971801fccccd1ff4a42593758c3b5aab3ab28a523"} Nov 02 13:43:18.973944 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:18.973915 2579 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="5d8b71e54bd055918f92701a1152fe4ad196f0e56dd4c54c41dbae2e877685cd" exitCode=0 Nov 02 13:43:18.974420 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:18.973953 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerDied","Data":"5d8b71e54bd055918f92701a1152fe4ad196f0e56dd4c54c41dbae2e877685cd"} Nov 02 13:43:19.719252 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:19.719163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:19.719252 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:19.719217 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:19.719437 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:19.719320 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:19.719437 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:19.719368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:19.719512 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:19.719449 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:19.719512 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:19.719494 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:19.979088 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:19.978999 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbv78" event={"ID":"16e35961-5e37-4e7b-bf8d-5a8d7c273893","Type":"ContainerStarted","Data":"5d3c59bcca63d63331342b147384d45b6f78d7a88dd9ff7e1e28fb88664c851c"} Nov 02 13:43:19.981809 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:19.981786 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:43:19.982150 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:19.982133 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"0fbc0ca5c6422e0270437b4c6d304e540fd68d07d5c5ca463930d43aee104069"} Nov 02 13:43:21.718708 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:21.718459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:21.718708 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:21.718483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:21.719268 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:21.718713 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:21.719268 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:21.718492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:21.719268 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:21.718781 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:21.719268 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:21.718867 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:21.990415 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:21.990344 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:43:21.990685 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:21.990664 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"eaedc35a523d7be7a97121c0f3d301b2eac01e04782af4268da3c172e7f3ce0f"} Nov 02 13:43:21.990965 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:21.990938 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:21.991147 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:21.991082 2579 scope.go:122] "RemoveContainer" containerID="d7a60ba7d2f5a5d78d0d248c98a0eeb8325a9f1105b2b849c6e46dc0d44fd514" Nov 02 13:43:22.007158 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:22.007133 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:22.098678 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:22.098628 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vbv78" podStartSLOduration=11.868003483 podStartE2EDuration="22.098614655s" totalImagesPullingTime="10.230611172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:20.071153969 +0000 UTC m=+19.866589390" watchObservedRunningTime="2026-11-02 13:43:22.098614655 +0000 UTC m=+21.894050075" Nov 02 13:43:22.313357 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:22.313273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:22.573757 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:22.573682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:22.996075 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:22.996047 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:43:22.996473 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:22.996397 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" event={"ID":"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a","Type":"ContainerStarted","Data":"02b38480cd41125e5cfffe002ec68a6ed8ab9fefbbc6d8a6c64705dfd7c72292"} Nov 02 13:43:22.996625 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:22.996599 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:22.996756 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:22.996682 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:23.011422 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.011401 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:23.162060 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.162007 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" podStartSLOduration=8.364173451 podStartE2EDuration="23.161971563s" totalImagesPullingTime="14.797798112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:23.085841797 +0000 UTC m=+22.881277218" watchObservedRunningTime="2026-11-02 13:43:23.161971563 +0000 UTC m=+22.957406984" Nov 02 13:43:23.487292 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.487261 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cc5c8"] Nov 02 13:43:23.487504 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.487388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:23.487504 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:23.487480 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:23.490518 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.490492 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-scx6x"] Nov 02 13:43:23.490616 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.490584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:23.490685 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:23.490665 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:23.491178 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.491159 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v8n7n"] Nov 02 13:43:23.491282 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.491269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:23.491405 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:23.491384 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:23.806317 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.806227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:23.978503 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:23.978470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:24.566573 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:24.566539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:24.782971 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:24.782935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:24.979292 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:24.979255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:25.149342 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:25.149310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:25.718654 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:25.718627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:25.719134 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:25.718718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:25.719134 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:25.718841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:25.719134 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:25.718857 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:25.719134 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:25.718933 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:25.719134 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:25.719026 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:25.873592 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:25.873502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:26.080301 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:26.080266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:26.807509 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:26.807413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:27.718768 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:27.718732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:27.718934 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:27.718733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:27.718934 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:27.718839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:27.718934 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:27.718835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:27.719118 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:27.718933 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:27.719118 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:27.719043 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:27.839728 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:27.839690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:28.037287 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:28.037191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:28.228823 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:28.228780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:28.463746 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:28.463712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:28.631510 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:28.631470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:28.858480 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:28.858399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:29.109088 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:29.109011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:29.412028 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:29.411822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:29.718862 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:29.718776 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:29.719041 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:29.718893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:29.719041 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:29.718907 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:29.719041 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:29.719020 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:29.719205 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:29.719079 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:29.719205 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:29.719137 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:29.849395 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:29.849359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:30.023636 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:30.023540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:30.214937 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:30.214887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:30.400506 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:30.400476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:30.635300 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:30.635266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:30.675659 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:30.675593 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9lm6_790cdff4-1780-4548-a327-e7b7df474347/dns-node-resolver/0.log" Nov 02 13:43:30.833962 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:30.833928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:31.054641 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.054571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:31.433854 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.433821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:31.678059 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.678033 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9d7n9_5291323c-ea92-499c-b246-a4f692f3f8c5/node-ca/0.log" Nov 02 13:43:31.718691 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.718609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:31.718691 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.718634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:31.718893 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.718713 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:31.718893 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:31.718730 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:31.718893 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:31.718778 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:31.718893 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:31.718847 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:31.731745 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.731726 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:31.731813 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.731806 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:43:31.732283 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.732266 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:43:31.807462 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:31.807425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:32.214865 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:32.214830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:32.383087 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:32.383054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:32.561942 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:32.561868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:32.811491 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:32.811461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:33.453423 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:33.453386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:33.453863 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:33.453436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:33.453863 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.453542 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:33.453863 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.453587 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:33.453863 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.453607 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret podName:a14e8823-8f98-40c4-b20d-dfea320c8304 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:05.453589388 +0000 UTC m=+65.249024802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret") pod "global-pull-secret-syncer-scx6x" (UID: "a14e8823-8f98-40c4-b20d-dfea320c8304") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:43:33.453863 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.453625 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs podName:88b783e5-08d6-44c6-ac37-b50adf8736ea nodeName:}" failed. No retries permitted until 2026-11-02 13:44:05.453613887 +0000 UTC m=+65.249049299 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs") pod "network-metrics-daemon-v8n7n" (UID: "88b783e5-08d6-44c6-ac37-b50adf8736ea") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:43:33.554528 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:33.554490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:33.554683 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.554642 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:43:33.554683 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.554656 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:43:33.554683 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.554665 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wvzgw for pod openshift-network-diagnostics/network-check-target-cc5c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:33.554800 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.554715 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw podName:9cb24650-63d2-4d50-9a20-db9bf4a5e2c9 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:05.554703971 +0000 UTC m=+65.350139383 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvzgw" (UniqueName: "kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw") pod "network-check-target-cc5c8" (UID: "9cb24650-63d2-4d50-9a20-db9bf4a5e2c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:43:33.714418 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:33.714389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:33.718365 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:33.718344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:33.718429 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:33.718410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:33.718469 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:33.718438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:33.718533 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.718515 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:33.718582 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.718555 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:33.718626 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:33.718617 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:33.913688 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:33.913653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:34.137095 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:34.137064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:34.338401 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:34.338315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:34.552732 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:34.552694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:34.815868 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:34.815822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:35.652244 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:35.652210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:35.718798 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:35.718760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:35.718798 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:35.718787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:35.719045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:35.718856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:35.719045 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:35.718870 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:35.719045 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:35.718933 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:35.719168 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:35.719042 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:36.027606 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:36.027529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:36.210184 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:36.210146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:36.410381 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:36.410348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:36.850554 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:36.850474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:37.430235 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:37.430203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:37.719187 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:37.719110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:37.719187 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:37.719135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:37.719187 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:37.719153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:37.719399 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:37.719239 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:37.719399 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:37.719365 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:37.719459 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:37.719432 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:38.010912 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:38.010827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:38.225350 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:38.225308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:38.414789 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:38.414758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:38.633023 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:38.632972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:38.971429 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:38.971402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:39.718474 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:39.718440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:39.718474 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:39.718476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:39.718915 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:39.718480 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:39.718915 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:39.718548 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:39.718915 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:39.718632 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:39.718915 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:39.718693 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:41.718322 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:41.718286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:41.718712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:41.718372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:41.718712 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:41.718461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:41.718712 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:41.718482 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:41.718712 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:41.718534 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:41.718712 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:41.718582 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:43.210658 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:43.210620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:43.409658 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:43.409622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:43.607694 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:43.607604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:43.718894 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:43.718867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:43.719067 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:43.718870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:43.719067 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:43.718971 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:43.719067 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:43.719052 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:43.719174 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:43.718870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:43.719174 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:43.719134 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:43.797976 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:43.797938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:44.013089 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:44.013052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:44.364797 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:44.364702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:44.538196 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:44.538164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:45.251997 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:45.251943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:45.719202 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:45.719172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:45.719636 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:45.719285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:45.719636 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:45.719294 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:45.719636 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:45.719310 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:45.719636 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:45.719371 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:45.719636 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:45.719509 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:47.719066 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:47.719030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:47.719462 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:47.719030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:47.719462 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:47.719144 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:47.719462 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:47.719030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:47.719462 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:47.719215 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:47.719462 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:47.719294 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:49.718310 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:49.718281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:49.718860 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:49.718365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:49.718860 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:49.718369 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5c8" podUID="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" Nov 02 13:43:49.718860 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:49.718417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:49.718860 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:49.718510 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scx6x" podUID="a14e8823-8f98-40c4-b20d-dfea320c8304" Nov 02 13:43:49.718860 ip-10-0-140-236 kubenswrapper[2579]: E1102 13:43:49.718627 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8n7n" podUID="88b783e5-08d6-44c6-ac37-b50adf8736ea" Nov 02 13:43:49.880447 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:49.880405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:50.141546 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.141508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:50.347879 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.347846 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:50.481970 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.481894 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeReady" Nov 02 13:43:50.482120 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.482035 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 13:43:50.563008 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.562968 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-87kgf"] Nov 02 13:43:50.568609 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.568585 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j79zx"] Nov 02 13:43:50.568961 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.568721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.570230 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.570208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:43:50.586039 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.586020 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 13:43:50.586220 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.586203 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 13:43:50.586285 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.586234 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wff6\"" Nov 02 13:43:50.586285 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.586235 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9sg29\"" Nov 02 13:43:50.588858 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.588845 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 13:43:50.589048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.589032 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 13:43:50.589115 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.589095 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 13:43:50.597714 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.597694 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 13:43:50.597714 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.597713 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 13:43:50.598839 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.598824 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j79zx"] Nov 02 13:43:50.599751 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.599728 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-87kgf"] Nov 02 13:43:50.675065 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.674473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cdde073d-a59a-4953-8650-3c997810c4b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.675065 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.674538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cdde073d-a59a-4953-8650-3c997810c4b9-crio-socket\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.675065 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.674596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef62365-5210-41a9-a45c-7e71cd04fb4c-cert\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:43:50.675065 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.674626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cdde073d-a59a-4953-8650-3c997810c4b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.675065 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.674646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z52s\" (UniqueName: \"kubernetes.io/projected/0ef62365-5210-41a9-a45c-7e71cd04fb4c-kube-api-access-9z52s\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:43:50.675065 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.674664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrgf\" (UniqueName: \"kubernetes.io/projected/cdde073d-a59a-4953-8650-3c997810c4b9-kube-api-access-drrgf\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.675065 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.674679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdde073d-a59a-4953-8650-3c997810c4b9-data-volume\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.675396 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.675293 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6bll8"] Nov 02 13:43:50.677102 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.677088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.691967 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.691948 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 13:43:50.692171 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.692157 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hkz22\"" Nov 02 13:43:50.692474 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.692462 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 13:43:50.703255 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.703234 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6bll8"] Nov 02 13:43:50.775458 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cdde073d-a59a-4953-8650-3c997810c4b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.775458 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z52s\" (UniqueName: \"kubernetes.io/projected/0ef62365-5210-41a9-a45c-7e71cd04fb4c-kube-api-access-9z52s\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:43:50.775458 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrgf\" (UniqueName: \"kubernetes.io/projected/cdde073d-a59a-4953-8650-3c997810c4b9-kube-api-access-drrgf\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/257163a1-1ead-438e-94c2-ca672bdcaa17-config-volume\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdde073d-a59a-4953-8650-3c997810c4b9-data-volume\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257163a1-1ead-438e-94c2-ca672bdcaa17-tmp-dir\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cdde073d-a59a-4953-8650-3c997810c4b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/257163a1-1ead-438e-94c2-ca672bdcaa17-metrics-tls\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cdde073d-a59a-4953-8650-3c997810c4b9-crio-socket\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx298\" (UniqueName: \"kubernetes.io/projected/257163a1-1ead-438e-94c2-ca672bdcaa17-kube-api-access-nx298\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775858 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cdde073d-a59a-4953-8650-3c997810c4b9-crio-socket\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.776038 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.775883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef62365-5210-41a9-a45c-7e71cd04fb4c-cert\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:43:50.776428 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.776045 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdde073d-a59a-4953-8650-3c997810c4b9-data-volume\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.776428 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.776239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cdde073d-a59a-4953-8650-3c997810c4b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.779630 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.779600 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cdde073d-a59a-4953-8650-3c997810c4b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.779725 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.779642 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef62365-5210-41a9-a45c-7e71cd04fb4c-cert\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:43:50.798853 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.798826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z52s\" (UniqueName: \"kubernetes.io/projected/0ef62365-5210-41a9-a45c-7e71cd04fb4c-kube-api-access-9z52s\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:43:50.798948 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.798864 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrgf\" (UniqueName: \"kubernetes.io/projected/cdde073d-a59a-4953-8650-3c997810c4b9-kube-api-access-drrgf\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.876725 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.876692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/257163a1-1ead-438e-94c2-ca672bdcaa17-config-volume\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.876725 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.876722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257163a1-1ead-438e-94c2-ca672bdcaa17-tmp-dir\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.876934 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.876741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/257163a1-1ead-438e-94c2-ca672bdcaa17-metrics-tls\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.876934 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.876878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx298\" (UniqueName: \"kubernetes.io/projected/257163a1-1ead-438e-94c2-ca672bdcaa17-kube-api-access-nx298\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.877063 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.877023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257163a1-1ead-438e-94c2-ca672bdcaa17-tmp-dir\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.877269 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.877245 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/257163a1-1ead-438e-94c2-ca672bdcaa17-config-volume\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.879034 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.879013 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/257163a1-1ead-438e-94c2-ca672bdcaa17-metrics-tls\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.883904 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.883885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:43:50.885948 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.885929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:43:50.900657 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.900631 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx298\" (UniqueName: \"kubernetes.io/projected/257163a1-1ead-438e-94c2-ca672bdcaa17-kube-api-access-nx298\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:50.955582 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.955549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:50.987872 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:50.987675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:51.219868 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.219840 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j79zx"] Nov 02 13:43:51.223181 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:51.223151 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef62365_5210_41a9_a45c_7e71cd04fb4c.slice/crio-20721192a61ad5faf3b6c895c5797a090afb06049309a20d9a5cc67939b4a3e0 WatchSource:0}: Error finding container 20721192a61ad5faf3b6c895c5797a090afb06049309a20d9a5cc67939b4a3e0: Status 404 returned error can't find the container with id 20721192a61ad5faf3b6c895c5797a090afb06049309a20d9a5cc67939b4a3e0 Nov 02 13:43:51.230143 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.230120 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-87kgf"] Nov 02 13:43:51.231708 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:51.231689 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdde073d_a59a_4953_8650_3c997810c4b9.slice/crio-0c49d4d28ce99735571e297417189ae9fca94ef8dd4923c7c6390df1f0daad08 WatchSource:0}: Error finding container 0c49d4d28ce99735571e297417189ae9fca94ef8dd4923c7c6390df1f0daad08: Status 404 returned error can't find the container with id 0c49d4d28ce99735571e297417189ae9fca94ef8dd4923c7c6390df1f0daad08 Nov 02 13:43:51.285668 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.285610 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6bll8"] Nov 02 13:43:51.288248 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:51.288225 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257163a1_1ead_438e_94c2_ca672bdcaa17.slice/crio-c0b381cdaaa15073f9edc1b62f135749f109532ba4839ac2d7fcdb6dffc0eabc WatchSource:0}: Error finding container c0b381cdaaa15073f9edc1b62f135749f109532ba4839ac2d7fcdb6dffc0eabc: Status 404 returned error can't find the container with id c0b381cdaaa15073f9edc1b62f135749f109532ba4839ac2d7fcdb6dffc0eabc Nov 02 13:43:51.718353 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.718317 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:43:51.718541 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.718389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:43:51.718541 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.718493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:43:51.733697 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.733398 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 13:43:51.733697 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.733505 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 13:43:51.734000 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.733959 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 13:43:51.734114 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.734068 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 13:43:51.734180 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.734141 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fs4v4\"" Nov 02 13:43:51.734237 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.734188 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lj29d\"" Nov 02 13:43:51.734237 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:51.734231 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xwmgj\"" Nov 02 13:43:52.058403 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:52.058205 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-87kgf" event={"ID":"cdde073d-a59a-4953-8650-3c997810c4b9","Type":"ContainerStarted","Data":"524f7445729c1d47958b8d6cf2de2370b6a048a13e956691bbf7af04228a8ca7"} Nov 02 13:43:52.058403 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:52.058310 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-87kgf" event={"ID":"cdde073d-a59a-4953-8650-3c997810c4b9","Type":"ContainerStarted","Data":"22cdc88092736e4f437d75c9a12bc291f9f4ae951427e0d6e6e3389fe31d1101"} Nov 02 13:43:52.058403 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:52.058331 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-87kgf" event={"ID":"cdde073d-a59a-4953-8650-3c997810c4b9","Type":"ContainerStarted","Data":"0c49d4d28ce99735571e297417189ae9fca94ef8dd4923c7c6390df1f0daad08"} Nov 02 13:43:52.060045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:52.060012 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j79zx" event={"ID":"0ef62365-5210-41a9-a45c-7e71cd04fb4c","Type":"ContainerStarted","Data":"20721192a61ad5faf3b6c895c5797a090afb06049309a20d9a5cc67939b4a3e0"} Nov 02 13:43:52.061620 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:52.061597 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bll8" event={"ID":"257163a1-1ead-438e-94c2-ca672bdcaa17","Type":"ContainerStarted","Data":"c0b381cdaaa15073f9edc1b62f135749f109532ba4839ac2d7fcdb6dffc0eabc"} Nov 02 13:43:52.427146 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:52.427116 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9lm6_790cdff4-1780-4548-a327-e7b7df474347/dns-node-resolver/0.log" Nov 02 13:43:53.253211 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.253149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:53.304944 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.304915 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p"] Nov 02 13:43:53.308050 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.308023 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.337269 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.337242 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 13:43:53.337434 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.337312 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 13:43:53.337434 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.337343 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 13:43:53.337434 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.337416 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fpdql\"" Nov 02 13:43:53.337584 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.337480 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 13:43:53.340314 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.340283 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p"] Nov 02 13:43:53.397694 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.397656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x4f\" (UniqueName: \"kubernetes.io/projected/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-kube-api-access-74x4f\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.397873 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.397708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.397873 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.397853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.397955 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.397904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.425954 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.425914 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 13:43:53.448366 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.448326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:53.498377 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.498350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.498497 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.498433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.498497 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.498487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x4f\" (UniqueName: \"kubernetes.io/projected/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-kube-api-access-74x4f\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.498572 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.498520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.499333 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.499305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.501266 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.501246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.501401 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.501379 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.522625 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.522595 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x4f\" (UniqueName: \"kubernetes.io/projected/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-kube-api-access-74x4f\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.615998 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.615950 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9d7n9_5291323c-ea92-499c-b246-a4f692f3f8c5/node-ca/0.log" Nov 02 13:43:53.621101 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.621075 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:43:53.628229 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.628200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:53.921094 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:53.921003 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p"] Nov 02 13:43:53.923971 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:53.923936 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6b7dd8_15b5_47b4_8fe7_b8a6d106cf35.slice/crio-8754eb72fb1713029ecad4fd8fb52c85dc14c5ee40aab2c9498793eb2a2e1f33 WatchSource:0}: Error finding container 8754eb72fb1713029ecad4fd8fb52c85dc14c5ee40aab2c9498793eb2a2e1f33: Status 404 returned error can't find the container with id 8754eb72fb1713029ecad4fd8fb52c85dc14c5ee40aab2c9498793eb2a2e1f33 Nov 02 13:43:54.068303 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.068270 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j79zx" event={"ID":"0ef62365-5210-41a9-a45c-7e71cd04fb4c","Type":"ContainerStarted","Data":"59c7032f7a475a4aa3a166248ec9b190a618710eea096c404fe6e9e502a50436"} Nov 02 13:43:54.070477 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.070450 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-87kgf" event={"ID":"cdde073d-a59a-4953-8650-3c997810c4b9","Type":"ContainerStarted","Data":"07ae1ae03d0f680721db1afd14ba908752f7d08bfe8cf9129c292afd4de0f233"} Nov 02 13:43:54.072430 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.072408 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bll8" event={"ID":"257163a1-1ead-438e-94c2-ca672bdcaa17","Type":"ContainerStarted","Data":"675703787f7158e8734b665b353deb3c70017c36ba74d52c27231b60abfcceb4"} Nov 02 13:43:54.072539 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.072452 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bll8" event={"ID":"257163a1-1ead-438e-94c2-ca672bdcaa17","Type":"ContainerStarted","Data":"2aab50b347aac979efe0c738387a88d40165e01c99cefa0d24e3a6fbc7c246d4"} Nov 02 13:43:54.072539 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.072518 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6bll8" Nov 02 13:43:54.073670 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.073646 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" event={"ID":"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35","Type":"ContainerStarted","Data":"8754eb72fb1713029ecad4fd8fb52c85dc14c5ee40aab2c9498793eb2a2e1f33"} Nov 02 13:43:54.109921 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.109862 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j79zx" podStartSLOduration=1.325362125 podStartE2EDuration="4.109847645s" totalImagesPullingTime="2.78448552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:54.10977827 +0000 UTC m=+53.905213692" watchObservedRunningTime="2026-11-02 13:43:54.109847645 +0000 UTC m=+53.905283065" Nov 02 13:43:54.151824 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.151775 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-6bll8" podStartSLOduration=2.333220936 podStartE2EDuration="4.151761723s" totalImagesPullingTime="1.818540787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:54.150886797 +0000 UTC m=+53.946322217" watchObservedRunningTime="2026-11-02 13:43:54.151761723 +0000 UTC m=+53.947197143" Nov 02 13:43:54.195003 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.194893 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-87kgf" podStartSLOduration=2.075781122 podStartE2EDuration="4.194881154s" totalImagesPullingTime="2.119100032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:54.193812984 +0000 UTC m=+53.989248405" watchObservedRunningTime="2026-11-02 13:43:54.194881154 +0000 UTC m=+53.990316574" Nov 02 13:43:54.423932 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.423905 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j79zx_0ef62365-5210-41a9-a45c-7e71cd04fb4c/serve-healthcheck-canary/0.log" Nov 02 13:43:54.542649 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.542570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:54.725068 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:54.725008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:55.013331 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:55.013304 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:43:56.080799 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:56.080760 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" event={"ID":"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35","Type":"ContainerStarted","Data":"2b83102f1ebb1ad926beea134a7d1b22c5db31b9f81f9d14a5c4fb644c9cc029"} Nov 02 13:43:56.081182 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:56.080824 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" event={"ID":"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35","Type":"ContainerStarted","Data":"4f457d1e63ed13ef46c176c6fa2f35afbec4643ba343d5f2163e42b80aaff0d3"} Nov 02 13:43:56.124334 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:56.124286 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" podStartSLOduration=1.946276923 podStartE2EDuration="3.12427199s" totalImagesPullingTime="1.177995067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:56.123013616 +0000 UTC m=+55.918449053" watchObservedRunningTime="2026-11-02 13:43:56.12427199 +0000 UTC m=+55.919707411" Nov 02 13:43:56.136569 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:56.136542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:56.523845 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:56.523804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:56.749277 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:56.749247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:57.210971 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:57.210933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:57.394936 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:57.394905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:57.735657 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:57.735627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:57.962695 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:57.962651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:58.048043 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.047962 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sjj7t"] Nov 02 13:43:58.050206 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.050190 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.083816 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.083791 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 13:43:58.083816 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.083810 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 13:43:58.084699 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.084682 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hnbmg\"" Nov 02 13:43:58.112649 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.112621 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 13:43:58.142960 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.142926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-wtmp\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.142960 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.142962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.143144 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.143075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6zn\" (UniqueName: \"kubernetes.io/projected/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-kube-api-access-4z6zn\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.143144 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.143111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-sys\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.143144 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.143134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-root\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.143230 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.143153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-textfile\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.143230 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.143198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-metrics-client-ca\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.143333 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.143249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-tls\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.143333 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.143267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.244614 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-metrics-client-ca\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-tls\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-wtmp\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6zn\" (UniqueName: \"kubernetes.io/projected/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-kube-api-access-4z6zn\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-sys\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-root\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-textfile\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244883 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-wtmp\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244952 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-root\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.244967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-sys\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245579 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.245204 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-textfile\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245579 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.245433 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-metrics-client-ca\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.245579 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.245454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.247116 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.247096 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-tls\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.247198 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.247120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.272530 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.272497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:58.280353 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.280334 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6zn\" (UniqueName: \"kubernetes.io/projected/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-kube-api-access-4z6zn\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.362049 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.361923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:43:58.371467 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:43:58.371435 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e273bc_9f51_4993_86c2_7e02c3b88c7d.slice/crio-782560dab0bfeb115690883ff55d5420c3f616779496c2516ee761093748e55c WatchSource:0}: Error finding container 782560dab0bfeb115690883ff55d5420c3f616779496c2516ee761093748e55c: Status 404 returned error can't find the container with id 782560dab0bfeb115690883ff55d5420c3f616779496c2516ee761093748e55c Nov 02 13:43:58.458768 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.458735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:58.631563 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.631530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:58.911572 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:58.911492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:59.091963 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:59.091925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:59.095705 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:59.095670 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjj7t" event={"ID":"d8e273bc-9f51-4993-86c2-7e02c3b88c7d","Type":"ContainerStarted","Data":"782560dab0bfeb115690883ff55d5420c3f616779496c2516ee761093748e55c"} Nov 02 13:43:59.293945 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:59.293913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:59.482048 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:59.481956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:59.687263 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:43:59.687233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:00.100658 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.100624 2579 generic.go:356] "Generic (PLEG): container finished" podID="d8e273bc-9f51-4993-86c2-7e02c3b88c7d" containerID="082261beb898c16786edd6810817530e6e3515c5ad03856fe181213860ab2deb" exitCode=0 Nov 02 13:44:00.100807 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.100693 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjj7t" event={"ID":"d8e273bc-9f51-4993-86c2-7e02c3b88c7d","Type":"ContainerDied","Data":"082261beb898c16786edd6810817530e6e3515c5ad03856fe181213860ab2deb"} Nov 02 13:44:00.160290 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.160265 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c6896987-j2hml"] Nov 02 13:44:00.163275 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.163256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.199919 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.199897 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 13:44:00.200060 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.200045 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 13:44:00.200250 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.200212 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 13:44:00.200250 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.200220 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 13:44:00.200250 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.200229 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tg4rm\"" Nov 02 13:44:00.200250 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.200238 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 13:44:00.200250 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.200225 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-16km2qrs03voi\"" Nov 02 13:44:00.207529 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.207500 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c6896987-j2hml"] Nov 02 13:44:00.259045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.259045 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-grpc-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.259288 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.259288 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98a7d07b-33d6-4778-8bb0-7c277468d407-metrics-client-ca\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.259288 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98a7d07b-33d6-4778-8bb0-7c277468d407-tmp\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.259444 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6l9\" (UniqueName: \"kubernetes.io/projected/98a7d07b-33d6-4778-8bb0-7c277468d407-kube-api-access-dg6l9\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.259444 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.259522 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.259567 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.259558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360415 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360415 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-grpc-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360415 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360947 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98a7d07b-33d6-4778-8bb0-7c277468d407-metrics-client-ca\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360947 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98a7d07b-33d6-4778-8bb0-7c277468d407-tmp\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360947 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6l9\" (UniqueName: \"kubernetes.io/projected/98a7d07b-33d6-4778-8bb0-7c277468d407-kube-api-access-dg6l9\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360947 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360947 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.360947 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.361279 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.360977 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98a7d07b-33d6-4778-8bb0-7c277468d407-tmp\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.361330 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.361288 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98a7d07b-33d6-4778-8bb0-7c277468d407-metrics-client-ca\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.363227 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.363197 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.363366 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.363325 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.363511 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.363486 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.363589 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.363527 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-grpc-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.363589 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.363532 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.363710 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.363689 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.385820 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.385795 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6l9\" (UniqueName: \"kubernetes.io/projected/98a7d07b-33d6-4778-8bb0-7c277468d407-kube-api-access-dg6l9\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.495351 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.495312 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:00.590085 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.589933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:00.781454 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.781416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:00.823952 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:00.823923 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c6896987-j2hml"] Nov 02 13:44:00.840025 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:44:00.839998 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7d07b_33d6_4778_8bb0_7c277468d407.slice/crio-0b4dbff4c5dfe453899913d18841cc6cce1d59da441d1b4c45b3767e487636a8 WatchSource:0}: Error finding container 0b4dbff4c5dfe453899913d18841cc6cce1d59da441d1b4c45b3767e487636a8: Status 404 returned error can't find the container with id 0b4dbff4c5dfe453899913d18841cc6cce1d59da441d1b4c45b3767e487636a8 Nov 02 13:44:01.104859 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:01.104780 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" event={"ID":"98a7d07b-33d6-4778-8bb0-7c277468d407","Type":"ContainerStarted","Data":"0b4dbff4c5dfe453899913d18841cc6cce1d59da441d1b4c45b3767e487636a8"} Nov 02 13:44:01.107051 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:01.107022 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjj7t" event={"ID":"d8e273bc-9f51-4993-86c2-7e02c3b88c7d","Type":"ContainerStarted","Data":"c87898a864de092ba50f3ff89b0894caa3c0473ab376f19387a1d798dda2e3b7"} Nov 02 13:44:01.107192 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:01.107066 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjj7t" event={"ID":"d8e273bc-9f51-4993-86c2-7e02c3b88c7d","Type":"ContainerStarted","Data":"7872bcdea8c18ddb4fbf0a0ba9f4fe29bc8b78327a321aec5255fcf1cd3da2fe"} Nov 02 13:44:01.154347 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:01.154296 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sjj7t" podStartSLOduration=2.265276237 podStartE2EDuration="3.154283514s" totalImagesPullingTime="822.056442ms" totalInitContainerRuntime="66.950835ms" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:01.154135783 +0000 UTC m=+60.949571204" watchObservedRunningTime="2026-11-02 13:44:01.154283514 +0000 UTC m=+60.949718935" Nov 02 13:44:01.311759 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:01.311722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:01.508152 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:01.508117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:01.754381 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:01.754338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:01.956997 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:01.956958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:02.778204 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.778182 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6"] Nov 02 13:44:02.780847 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.780826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:02.798545 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.798521 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cv5n8\"" Nov 02 13:44:02.807477 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.807434 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 13:44:02.811358 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.811333 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6"] Nov 02 13:44:02.843329 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.843290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:02.881191 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.881165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmcr\" (UniqueName: \"kubernetes.io/projected/d5655095-e60c-40a9-9f0d-510e96cdd667-kube-api-access-tvmcr\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:02.881391 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.881222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5655095-e60c-40a9-9f0d-510e96cdd667-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:02.982228 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.982196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmcr\" (UniqueName: \"kubernetes.io/projected/d5655095-e60c-40a9-9f0d-510e96cdd667-kube-api-access-tvmcr\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:02.982228 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.982242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5655095-e60c-40a9-9f0d-510e96cdd667-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:02.984610 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:02.984582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5655095-e60c-40a9-9f0d-510e96cdd667-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:03.007408 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:03.007369 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmcr\" (UniqueName: \"kubernetes.io/projected/d5655095-e60c-40a9-9f0d-510e96cdd667-kube-api-access-tvmcr\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:03.027557 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:03.027481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:03.110532 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:03.110502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:03.117821 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:03.117790 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" event={"ID":"98a7d07b-33d6-4778-8bb0-7c277468d407","Type":"ContainerStarted","Data":"14621580593b246b7e61f92c6c5cd297d39960847bc1d285ed254b6936fcb5ba"} Nov 02 13:44:03.117944 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:03.117856 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" event={"ID":"98a7d07b-33d6-4778-8bb0-7c277468d407","Type":"ContainerStarted","Data":"0dc86561a1bdc674124971c2b890ef2bda07b5c61ef3c51c8e42dd4e3e3cc127"} Nov 02 13:44:03.117944 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:03.117883 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" event={"ID":"98a7d07b-33d6-4778-8bb0-7c277468d407","Type":"ContainerStarted","Data":"1f0afedf3c68a5bbbb7f990fd6fe3617b76800ac54b42fc9588cc3467273b36d"} Nov 02 13:44:03.257217 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:03.257182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:03.419537 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:03.419507 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6"] Nov 02 13:44:03.422223 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:44:03.422189 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5655095_e60c_40a9_9f0d_510e96cdd667.slice/crio-b17ed04db612c0c3334014fe1a12751eadf16567f191474709612605d65f0351 WatchSource:0}: Error finding container b17ed04db612c0c3334014fe1a12751eadf16567f191474709612605d65f0351: Status 404 returned error can't find the container with id b17ed04db612c0c3334014fe1a12751eadf16567f191474709612605d65f0351 Nov 02 13:44:04.078638 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:04.078561 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:04.124117 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:04.124077 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" event={"ID":"98a7d07b-33d6-4778-8bb0-7c277468d407","Type":"ContainerStarted","Data":"3ae320335552a44d9a9686fc450310548f4dad965f1370ae32653d1f31053ce2"} Nov 02 13:44:04.124290 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:04.124177 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" event={"ID":"98a7d07b-33d6-4778-8bb0-7c277468d407","Type":"ContainerStarted","Data":"3e5659cd60dda8527017db168a4c06b6bddc631447531b63af16ef0fc04f3c94"} Nov 02 13:44:04.124290 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:04.124217 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" event={"ID":"98a7d07b-33d6-4778-8bb0-7c277468d407","Type":"ContainerStarted","Data":"0fbbc2418d35af5fd6eeb78f74c7fcd36b561dffa44977ad10cbfccfe16cd282"} Nov 02 13:44:04.124290 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:04.124250 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:04.125291 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:04.125269 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" event={"ID":"d5655095-e60c-40a9-9f0d-510e96cdd667","Type":"ContainerStarted","Data":"b17ed04db612c0c3334014fe1a12751eadf16567f191474709612605d65f0351"} Nov 02 13:44:04.175637 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:04.175581 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" podStartSLOduration=1.4291364 podStartE2EDuration="4.175565221s" totalImagesPullingTime="2.746428821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:04.175306083 +0000 UTC m=+63.970741506" watchObservedRunningTime="2026-11-02 13:44:04.175565221 +0000 UTC m=+63.971000643" Nov 02 13:44:05.071331 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.071241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:05.130207 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.130165 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" event={"ID":"d5655095-e60c-40a9-9f0d-510e96cdd667","Type":"ContainerStarted","Data":"4cc4d60adbebefebf51b4ebbab58ac0725405ce520926d2de7e0050c4feac61e"} Nov 02 13:44:05.130635 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.130404 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:05.135432 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.135411 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:05.180118 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.180073 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" podStartSLOduration=1.8656109459999999 podStartE2EDuration="3.180058469s" totalImagesPullingTime="1.314447523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:05.179619772 +0000 UTC m=+64.975055193" watchObservedRunningTime="2026-11-02 13:44:05.180058469 +0000 UTC m=+64.975493890" Nov 02 13:44:05.506688 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.506644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:05.506878 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.506698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:44:05.520732 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.520686 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 13:44:05.520732 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.520715 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 13:44:05.531071 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.530586 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:05.531963 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.531931 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:44:05.545881 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.545857 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lj29d\"" Nov 02 13:44:05.553041 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.553023 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:44:05.560441 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.560423 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fs4v4\"" Nov 02 13:44:05.567786 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.567762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:05.611300 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.607329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:44:05.621253 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.621220 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 13:44:05.641339 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.641311 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 13:44:05.650791 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.650764 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:44:05.853433 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.853353 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xwmgj\"" Nov 02 13:44:05.857519 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.857493 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v8n7n"] Nov 02 13:44:05.862008 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.860583 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:44:05.876800 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:05.876765 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-scx6x"] Nov 02 13:44:05.879777 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:44:05.879751 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14e8823_8f98_40c4_b20d_dfea320c8304.slice/crio-bd9f1c9b783cdf5120ff0e063a238cc5dc52425e2f6ddeba4b5d77eabad36e80 WatchSource:0}: Error finding container bd9f1c9b783cdf5120ff0e063a238cc5dc52425e2f6ddeba4b5d77eabad36e80: Status 404 returned error can't find the container with id bd9f1c9b783cdf5120ff0e063a238cc5dc52425e2f6ddeba4b5d77eabad36e80 Nov 02 13:44:06.136690 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:06.136654 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-scx6x" event={"ID":"a14e8823-8f98-40c4-b20d-dfea320c8304","Type":"ContainerStarted","Data":"bd9f1c9b783cdf5120ff0e063a238cc5dc52425e2f6ddeba4b5d77eabad36e80"} Nov 02 13:44:06.137814 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:06.137786 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v8n7n" event={"ID":"88b783e5-08d6-44c6-ac37-b50adf8736ea","Type":"ContainerStarted","Data":"8ba2e9dcaade4fd7469acc78b70fb7d03a3a3138d118f3d16b033720ea23d62c"} Nov 02 13:44:06.162861 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:06.162833 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cc5c8"] Nov 02 13:44:06.165682 ip-10-0-140-236 kubenswrapper[2579]: W1102 13:44:06.165659 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb24650_63d2_4d50_9a20_db9bf4a5e2c9.slice/crio-9aa2152568f8536cf170e981fa4df2945208c1d84af2097b5e997667d4422530 WatchSource:0}: Error finding container 9aa2152568f8536cf170e981fa4df2945208c1d84af2097b5e997667d4422530: Status 404 returned error can't find the container with id 9aa2152568f8536cf170e981fa4df2945208c1d84af2097b5e997667d4422530 Nov 02 13:44:07.142701 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:07.142310 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cc5c8" event={"ID":"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9","Type":"ContainerStarted","Data":"9aa2152568f8536cf170e981fa4df2945208c1d84af2097b5e997667d4422530"} Nov 02 13:44:07.145520 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:07.145493 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v8n7n" event={"ID":"88b783e5-08d6-44c6-ac37-b50adf8736ea","Type":"ContainerStarted","Data":"a89b1d9318285e65a6ae8cf260de390600f31979baf912be07eda829e67e9a2d"} Nov 02 13:44:07.769625 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:07.769588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:07.952635 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:07.952554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:08.150654 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:08.150620 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v8n7n" event={"ID":"88b783e5-08d6-44c6-ac37-b50adf8736ea","Type":"ContainerStarted","Data":"b7d8f26cc52a485062956fca2d5bba5ad7a5524cdac68d448b5116dc493f1f4b"} Nov 02 13:44:08.192454 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:08.192393 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v8n7n" podStartSLOduration=67.105056464 podStartE2EDuration="1m8.192373756s" totalImagesPullingTime="1.087317292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:08.190775735 +0000 UTC m=+67.986211156" watchObservedRunningTime="2026-11-02 13:44:08.192373756 +0000 UTC m=+67.987809178" Nov 02 13:44:10.137342 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:10.137313 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:10.160929 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:10.160899 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cc5c8" event={"ID":"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9","Type":"ContainerStarted","Data":"ede265df4275f7aaa286429507b46b3cd499a975b6c1611e725ec4cf145c510c"} Nov 02 13:44:10.161115 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:10.160973 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:44:10.262392 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:10.262324 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cc5c8" podStartSLOduration=66.942235958 podStartE2EDuration="1m10.262304363s" totalImagesPullingTime="3.320068405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:10.260686519 +0000 UTC m=+70.056121941" watchObservedRunningTime="2026-11-02 13:44:10.262304363 +0000 UTC m=+70.057739785" Nov 02 13:44:10.775313 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:10.775244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:10.959550 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:10.959490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:11.169032 ip-10-0-140-236 kubenswrapper[2579]: I1102 13:44:11.168999 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-scx6x" event={"ID":"a14e8823-8f98-40c4-b20d-dfea320c8304","Type":"ContainerStarted","Data":"b57a1d2ccb0c3f4bc899304abef6bea72420299471c411013ec1215ee63a00f0"} Nov 02 13:44:11.189338 ip-10-0-140-236 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 13:44:11.209549 ip-10-0-140-236 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 13:44:11.209810 ip-10-0-140-236 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 13:44:11.209877 ip-10-0-140-236 systemd[1]: kubelet.service: Consumed 3.949s CPU time. Nov 02 13:44:11.223849 ip-10-0-140-236 systemd[1]: Starting Kubernetes Kubelet... Nov 02 13:44:11.313124 ip-10-0-140-236 kubenswrapper[5591]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:44:11.313124 ip-10-0-140-236 kubenswrapper[5591]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 13:44:11.313124 ip-10-0-140-236 kubenswrapper[5591]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:44:11.313124 ip-10-0-140-236 kubenswrapper[5591]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316244 5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316358 5591 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316364 5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316369 5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316374 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316380 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316389 5591 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316395 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316400 5591 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316405 5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316410 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316416 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316421 5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316426 5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316431 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316435 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316440 5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316445 5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316450 5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:44:11.327479 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316455 5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316459 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316464 5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316469 5591 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316475 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316479 5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316485 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316490 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316495 5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316500 5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316505 5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316510 5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316515 5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316519 5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316525 5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316530 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316534 5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316539 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316544 5591 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:44:11.328284 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316549 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316555 5591 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316560 5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316565 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316570 5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316575 5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316580 5591 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316586 5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316591 5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316596 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316600 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316605 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316610 5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316615 5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316620 5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316624 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316629 5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316634 5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316639 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316643 5591 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:44:11.329164 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316648 5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316653 5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316659 5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316664 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316672 5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316682 5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316688 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316695 5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316700 5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316705 5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316710 5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316715 5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316720 5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316726 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316732 5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316736 5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316743 5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316750 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316755 5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:44:11.329941 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316761 5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316765 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316770 5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316774 5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316779 5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316783 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316789 5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316793 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316799 5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316804 5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316807 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316812 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316818 5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316823 5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316827 5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316832 5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316838 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316842 5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316850 5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:44:11.330862 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316855 5591 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316859 5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316865 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316869 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316874 5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316879 5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316883 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316888 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316893 5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.316907 5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318238 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318250 5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318256 5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318260 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318265 5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318269 5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318274 5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318279 5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318286 5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318291 5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:44:11.331629 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318295 5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318300 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318305 5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318309 5591 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318314 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318318 5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318323 5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318328 5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318333 5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318338 5591 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318342 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318350 5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318355 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318360 5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318365 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318369 5591 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318374 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318378 5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318383 5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318387 5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:44:11.332395 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318392 5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318396 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318409 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318413 5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318418 5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318426 5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318434 5591 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318440 5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318445 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318450 5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318455 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318460 5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318467 5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318473 5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318479 5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318483 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318488 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318492 5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318497 5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:44:11.333212 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318502 5591 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318507 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318511 5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318515 5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318522 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318527 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318531 5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318536 5591 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318541 5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318546 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318551 5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318555 5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318560 5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318564 5591 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318569 5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318573 5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318579 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318584 5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318589 5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318593 5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:44:11.333937 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318597 5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318601 5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318606 5591 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318610 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318615 5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318621 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318625 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318630 5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318634 5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318639 5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318643 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318648 5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318652 5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318657 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318661 5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318666 5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318673 5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318678 5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318683 5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:44:11.334703 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318687 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318692 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318696 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318701 5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318706 5591 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318710 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318715 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318719 5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318724 5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318728 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318744 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318749 5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318754 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318758 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318763 5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318768 5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318773 5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.318780 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319746 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:44:11.335191 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319759 5591 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319765 5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319770 5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319774 5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319779 5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319783 5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319787 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319791 5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319795 5591 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319800 5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319807 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319812 5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319816 5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319821 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319825 5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319832 5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319839 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319844 5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319848 5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:44:11.335689 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319853 5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319862 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319867 5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319872 5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319878 5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319882 5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319892 5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319898 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319904 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319909 5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319915 5591 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319920 5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319924 5591 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319930 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319934 5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319938 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319942 5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319947 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319951 5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:44:11.336444 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319956 5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319961 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319966 5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319970 5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.319976 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320008 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320014 5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320019 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320023 5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320028 5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320034 5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320038 5591 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320043 5591 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320048 5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320053 5591 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320058 5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320064 5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320069 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320075 5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320081 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:44:11.336902 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320086 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320091 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320096 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320100 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320105 5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320110 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320114 5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320118 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320123 5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320127 5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320131 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320136 5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320141 5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320145 5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320150 5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320155 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320163 5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320167 5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320172 5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:44:11.337441 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320176 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320181 5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320185 5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320190 5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320194 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320198 5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320203 5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320208 5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320212 5591 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320218 5591 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320222 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320227 5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320234 5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320239 5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320244 5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320250 5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320254 5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320259 5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320264 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320269 5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:44:11.337888 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320273 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320278 5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320282 5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320287 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320291 5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320296 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320300 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.320304 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320502 5591 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320523 5591 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320538 5591 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320546 5591 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320553 5591 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320558 5591 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320565 5591 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320573 5591 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320578 5591 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320584 5591 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320590 5591 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320595 5591 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320600 5591 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320605 5591 flags.go:64] FLAG: --cgroup-root="" Nov 02 13:44:11.338486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320610 5591 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320616 5591 flags.go:64] FLAG: --client-ca-file="" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320628 5591 flags.go:64] FLAG: --cloud-provider="external" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320635 5591 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320643 5591 flags.go:64] FLAG: --cluster-domain="" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320648 5591 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320654 5591 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320659 5591 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320665 5591 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320672 5591 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320677 5591 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320682 5591 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320688 5591 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320694 5591 flags.go:64] FLAG: --contention-profiling="false" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320699 5591 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320705 5591 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320711 5591 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320716 5591 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320723 5591 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320732 5591 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320737 5591 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320743 5591 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320748 5591 flags.go:64] FLAG: --enable-server="true" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320755 5591 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320761 5591 flags.go:64] FLAG: --event-burst="100" Nov 02 13:44:11.341236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320767 5591 flags.go:64] FLAG: --event-qps="50" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320772 5591 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320778 5591 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320784 5591 flags.go:64] FLAG: --eviction-hard="" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320790 5591 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320795 5591 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320800 5591 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320806 5591 flags.go:64] FLAG: --eviction-soft="" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320811 5591 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320818 5591 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320824 5591 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320830 5591 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320835 5591 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320840 5591 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320845 5591 flags.go:64] FLAG: --feature-gates="" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320852 5591 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320858 5591 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320864 5591 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320870 5591 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320876 5591 flags.go:64] FLAG: --healthz-port="10248" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320882 5591 flags.go:64] FLAG: --help="false" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320886 5591 flags.go:64] FLAG: --hostname-override="ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320892 5591 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320898 5591 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 13:44:11.342046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320903 5591 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320909 5591 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320917 5591 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320923 5591 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320928 5591 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320934 5591 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320940 5591 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320945 5591 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320952 5591 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320957 5591 flags.go:64] FLAG: --kube-reserved="" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320962 5591 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320966 5591 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320971 5591 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.320977 5591 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321022 5591 flags.go:64] FLAG: --lock-file="" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321028 5591 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321034 5591 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321042 5591 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321052 5591 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321057 5591 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321063 5591 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321068 5591 flags.go:64] FLAG: --logging-format="text" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321073 5591 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321080 5591 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 13:44:11.342704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321086 5591 flags.go:64] FLAG: --manifest-url="" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321090 5591 flags.go:64] FLAG: --manifest-url-header="" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321098 5591 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321103 5591 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321110 5591 flags.go:64] FLAG: --max-pods="110" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321116 5591 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321121 5591 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321127 5591 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321133 5591 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321138 5591 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321145 5591 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321150 5591 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321163 5591 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321169 5591 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321174 5591 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321179 5591 flags.go:64] FLAG: --pod-cidr="" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321185 5591 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321190 5591 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321196 5591 flags.go:64] FLAG: --pods-per-core="0" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321201 5591 flags.go:64] FLAG: --port="10250" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321207 5591 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321212 5591 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ae846ada92c766e9" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321217 5591 flags.go:64] FLAG: --qos-reserved="" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321222 5591 flags.go:64] FLAG: --read-only-port="10255" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321228 5591 flags.go:64] FLAG: --register-node="true" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321236 5591 flags.go:64] FLAG: --register-with-taints="" Nov 02 13:44:11.343400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321243 5591 flags.go:64] FLAG: --registry-burst="10" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321249 5591 flags.go:64] FLAG: --registry-qps="5" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321255 5591 flags.go:64] FLAG: --reserved-cpus="" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321260 5591 flags.go:64] FLAG: --reserved-memory="" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321268 5591 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321273 5591 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321279 5591 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321284 5591 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321290 5591 flags.go:64] FLAG: --runonce="false" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321295 5591 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321300 5591 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321305 5591 flags.go:64] FLAG: --seccomp-default="false" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321311 5591 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321317 5591 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321323 5591 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321328 5591 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321335 5591 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321340 5591 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321346 5591 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321351 5591 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321356 5591 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321362 5591 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321368 5591 flags.go:64] FLAG: --system-cgroups="" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321373 5591 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321383 5591 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 13:44:11.344112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321389 5591 flags.go:64] FLAG: --tls-cert-file="" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321393 5591 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321402 5591 flags.go:64] FLAG: --tls-min-version="" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321407 5591 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321412 5591 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321417 5591 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321424 5591 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321431 5591 flags.go:64] FLAG: --v="2" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321438 5591 flags.go:64] FLAG: --version="false" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321446 5591 flags.go:64] FLAG: --vmodule="" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321452 5591 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.321459 5591 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321705 5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321716 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321722 5591 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321728 5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321732 5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321737 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321741 5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321745 5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321751 5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321756 5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321761 5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:44:11.344746 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321768 5591 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321772 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321777 5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321781 5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321787 5591 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321791 5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321796 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321800 5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321805 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321810 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321814 5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321819 5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321823 5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321828 5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321833 5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321840 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321844 5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321850 5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321855 5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:44:11.345329 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321859 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321865 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321870 5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321875 5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321879 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321883 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321888 5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321892 5591 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321897 5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321904 5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321911 5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321917 5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321922 5591 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321930 5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321935 5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321939 5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321944 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321948 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321953 5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321958 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:44:11.345777 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321962 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321967 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321972 5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321977 5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.321999 5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322003 5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322008 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322012 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322022 5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322028 5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322033 5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322038 5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322043 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322047 5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322052 5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322058 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322063 5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322067 5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322073 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:44:11.346484 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322077 5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322082 5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322086 5591 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322091 5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322095 5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322099 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322106 5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322111 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322116 5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322121 5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322125 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322130 5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322134 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322138 5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322143 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322147 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322152 5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322157 5591 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322161 5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:44:11.347039 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322166 5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322171 5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322177 5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322183 5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322187 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322191 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322196 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322201 5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322206 5591 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322211 5591 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322216 5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322220 5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322225 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322231 5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322236 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322240 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322245 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:44:11.347569 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322249 5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.322299 5591 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322358 5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322368 5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322374 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322379 5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322383 5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322389 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322394 5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322399 5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322404 5591 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322409 5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322413 5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322418 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322422 5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322427 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:44:11.348057 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322431 5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322438 5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322444 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322448 5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322452 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322457 5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322462 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322467 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322471 5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322476 5591 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322482 5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322486 5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322491 5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322496 5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322501 5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322505 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322510 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322515 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322519 5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322526 5591 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:44:11.348471 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322531 5591 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322535 5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322540 5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322545 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322550 5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322555 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322560 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322564 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322569 5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322574 5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322578 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322583 5591 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322588 5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322594 5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322599 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322604 5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322608 5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322614 5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322618 5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322623 5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:44:11.348969 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322628 5591 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322632 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322637 5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322642 5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322647 5591 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322652 5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322657 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322663 5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322668 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322672 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322677 5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322686 5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322690 5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322696 5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322700 5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322705 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322709 5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322714 5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322719 5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:44:11.349529 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322723 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322730 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322735 5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322739 5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322744 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322749 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322756 5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322761 5591 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322766 5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322774 5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322780 5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322785 5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322790 5591 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322795 5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322800 5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322804 5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322809 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322814 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322818 5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322822 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:44:11.350179 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322827 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322832 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322836 5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322841 5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322848 5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322853 5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322858 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322862 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322869 5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322877 5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322882 5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322887 5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.322892 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.328690 5591 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.328709 5591 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.328734 5591 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.328741 5591 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328855 5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328863 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:44:11.350721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328869 5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328875 5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328880 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328884 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328889 5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328893 5591 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328897 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328901 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328906 5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328910 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328921 5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328927 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328931 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328935 5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328939 5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328943 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328948 5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328954 5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328959 5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:44:11.351211 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328963 5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328967 5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328971 5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328975 5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.328996 5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329001 5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329005 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329010 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329014 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329019 5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329023 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329028 5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329032 5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329036 5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329040 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329045 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329049 5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329054 5591 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329058 5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329062 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:44:11.351668 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329067 5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329071 5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329075 5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329080 5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329093 5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329098 5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329103 5591 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329107 5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329111 5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329115 5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329121 5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329126 5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329130 5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329134 5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329139 5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329143 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329147 5591 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329152 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329156 5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:44:11.352282 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329161 5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329165 5591 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329170 5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329174 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329178 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329183 5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329188 5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329192 5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329197 5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329201 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329205 5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329209 5591 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329214 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329218 5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329222 5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329225 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329230 5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329233 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329244 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:44:11.352810 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329248 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329254 5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329260 5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329265 5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329271 5591 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329275 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329279 5591 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329283 5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329287 5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329292 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329295 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329299 5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329303 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329308 5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329312 5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329316 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329320 5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329324 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329331 5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:44:11.353274 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329337 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329341 5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329345 5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329350 5591 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329354 5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329358 5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329363 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329367 5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.329410 5591 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329674 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329681 5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329685 5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329689 5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329701 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329706 5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329710 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:44:11.353968 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329715 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329719 5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329725 5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329729 5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329733 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329737 5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329741 5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329745 5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329749 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329753 5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329758 5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329762 5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329766 5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329770 5591 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329774 5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329778 5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329783 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329787 5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329791 5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:44:11.354635 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329795 5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329799 5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329803 5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329808 5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329812 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329816 5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329820 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329824 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329828 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329833 5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329839 5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329851 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329856 5591 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329860 5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329865 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329870 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329874 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329878 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329883 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:44:11.355468 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329887 5591 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329890 5591 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329895 5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329898 5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329904 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329908 5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329912 5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329916 5591 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329920 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329924 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329928 5591 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329932 5591 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329936 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329940 5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329944 5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329948 5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329953 5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329957 5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329961 5591 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329965 5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:44:11.356223 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329970 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.329974 5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330057 5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330065 5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330070 5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330088 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330093 5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330097 5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330103 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330107 5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330112 5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330116 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330121 5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330125 5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330129 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330133 5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330139 5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330143 5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330147 5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:44:11.357041 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330151 5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330155 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330159 5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330163 5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330167 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330172 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330176 5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330180 5591 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330184 5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330195 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330199 5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330203 5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330207 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330211 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330215 5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330219 5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330223 5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330227 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330231 5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330241 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:44:11.357602 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330245 5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330250 5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.330295 5591 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330361 5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330368 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330374 5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330380 5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330387 5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330393 5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330399 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330403 5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330408 5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330411 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330416 5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330420 5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330424 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:44:11.358343 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330428 5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330433 5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330437 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330441 5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330445 5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330450 5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330454 5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330459 5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330463 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330467 5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330472 5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330476 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330480 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330484 5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330488 5591 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330493 5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330506 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330511 5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330518 5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330523 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:44:11.359006 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330528 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330533 5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330538 5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330542 5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330547 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330553 5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330558 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330563 5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330568 5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330573 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330578 5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330583 5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330589 5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330594 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330598 5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330605 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330609 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330614 5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330619 5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:44:11.359727 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330623 5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330628 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330632 5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330636 5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330641 5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330645 5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330649 5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330654 5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330658 5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330662 5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330673 5591 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330679 5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330684 5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330688 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330692 5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330697 5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330701 5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330705 5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330712 5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:44:11.360489 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330716 5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330721 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330725 5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330729 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330733 5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330737 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330743 5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330749 5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330753 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330758 5591 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330762 5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330766 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330770 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330774 5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330778 5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330783 5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330787 5591 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330791 5591 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330795 5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:44:11.361341 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330799 5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330803 5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330807 5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330811 5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330815 5591 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330827 5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330833 5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330837 5591 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330841 5591 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330845 5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330849 5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330852 5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330857 5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330861 5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330865 5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:11.330869 5591 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.330949 5591 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.332173 5591 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.332247 5591 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.332697 5591 server.go:1014] "Starting client certificate rotation" Nov 02 13:44:11.362309 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.333001 5591 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:44:11.363205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.333060 5591 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:38:00 +0000 UTC" deadline="2028-07-23 11:39:21.276789174 +0000 UTC" Nov 02 13:44:11.363205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.333095 5591 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15093h55m9.943697137s" Nov 02 13:44:11.363205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.334692 5591 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:44:11.363205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.338080 5591 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:44:11.363205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.339444 5591 log.go:25] "Validated CRI v1 runtime API" Nov 02 13:44:11.363205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.341272 5591 log.go:25] "Validated CRI v1 image API" Nov 02 13:44:11.363205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.342302 5591 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 13:44:11.363205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.344294 5591 fs.go:135] Filesystem UUIDs: map[13807ce8-e841-409d-97c5-2405b69749da:/dev/nvme0n1p3 4f775794-7b44-418a-a8a5-312c5dc5500f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 02 13:44:11.363504 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.344322 5591 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b4dbff4c5dfe453899913d18841cc6cce1d59da441d1b4c45b3767e487636a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b4dbff4c5dfe453899913d18841cc6cce1d59da441d1b4c45b3767e487636a8/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c49d4d28ce99735571e297417189ae9fca94ef8dd4923c7c6390df1f0daad08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c49d4d28ce99735571e297417189ae9fca94ef8dd4923c7c6390df1f0daad08/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e0d4d8a8a7c4fa08c9e19f47c0a644c400e19336a7a83c887d29deeebebaf4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e0d4d8a8a7c4fa08c9e19f47c0a644c400e19336a7a83c887d29deeebebaf4d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12ba0dc43d43f4899ac81c7c60b103ec4137895ff6c963087c531e717d539d56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12ba0dc43d43f4899ac81c7c60b103ec4137895ff6c963087c531e717d539d56/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d7ca799ea3892c4ea1b4495d403cc484b6e5f5d05d7f529f286b04698d77783/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d7ca799ea3892c4ea1b4495d403cc484b6e5f5d05d7f529f286b04698d77783/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20721192a61ad5faf3b6c895c5797a090afb06049309a20d9a5cc67939b4a3e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20721192a61ad5faf3b6c895c5797a090afb06049309a20d9a5cc67939b4a3e0/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c484b97bf8b5f0634d95075477c05abe6a73beb822241db51b0421b2152e197/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c484b97bf8b5f0634d95075477c05abe6a73beb822241db51b0421b2152e197/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6caba99e97540d491261dd7a170d10ff675ac0a6ccc14a1feef24fb6e9600bdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6caba99e97540d491261dd7a170d10ff675ac0a6ccc14a1feef24fb6e9600bdb/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71fbdaae8e4207f512c28ea54b6db30c72bc3a0c775210fba12a1f0586e655ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71fbdaae8e4207f512c28ea54b6db30c72bc3a0c775210fba12a1f0586e655ba/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/782560dab0bfeb115690883ff55d5420c3f616779496c2516ee761093748e55c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/782560dab0bfeb115690883ff55d5420c3f616779496c2516ee761093748e55c/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80565425f70ad633a65bab7ce1d5af11122c5830bb78de1737297d3b4a7a8dcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80565425f70ad633a65bab7ce1d5af11122c5830bb78de1737297d3b4a7a8dcd/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8754eb72fb1713029ecad4fd8fb52c85dc14c5ee40aab2c9498793eb2a2e1f33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8754eb72fb1713029ecad4fd8fb52c85dc14c5ee40aab2c9498793eb2a2e1f33/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ba2e9dcaade4fd7469acc78b70fb7d03a3a3138d118f3d16b033720ea23d62c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ba2e9dcaade4fd7469acc78b70fb7d03a3a3138d118f3d16b033720ea23d62c/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9aa2152568f8536cf170e981fa4df2945208c1d84af2097b5e997667d4422530/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9aa2152568f8536cf170e981fa4df2945208c1d84af2097b5e997667d4422530/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b17ed04db612c0c3334014fe1a12751eadf16567f191474709612605d65f0351/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b17ed04db612c0c3334014fe1a12751eadf16567f191474709612605d65f0351/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7688829976fff75a5c9900b8ae22fb29699368757ec7d6cba1610fb787f5f03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7688829976fff75a5c9900b8ae22fb29699368757ec7d6cba1610fb787f5f03/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd9f1c9b783cdf5120ff0e063a238cc5dc52425e2f6ddeba4b5d77eabad36e80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd9f1c9b783cdf5120ff0e063a238cc5dc52425e2f6ddeba4b5d77eabad36e80/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0b381cdaaa15073f9edc1b62f135749f109532ba4839ac2d7fcdb6dffc0eabc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0b381cdaaa15073f9edc1b62f135749f109532ba4839ac2d7fcdb6dffc0eabc/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccb8ebaf7df6a5841a53771b5ebfd0caf034c96f71b441bb9d37a1605e28906c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccb8ebaf7df6a5841a53771b5ebfd0caf034c96f71b441bb9d37a1605e28906c/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2e469850889f5454ef32b7c2a7741f5a31573051185b3e7a5e8778b2874f7f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2e469850889f5454ef32b7c2a7741f5a31573051185b3e7a5e8778b2874f7f4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ef62365-5210-41a9-a45c-7e71cd04fb4c/volumes/kubernetes.io~projected/kube-api-access-9z52s:{mountpoint:/var/lib/kubelet/pods/0ef62365-5210-41a9-a45c-7e71cd04fb4c/volumes/kubernetes.io~projected/kube-api-access-9z52s major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ef62365-5210-41a9-a45c-7e71cd04fb4c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0ef62365-5210-41a9-a45c-7e71cd04fb4c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16e35961-5e37-4e7b-bf8d-5a8d7c273893/volumes/kubernetes.io~projected/kube-api-access-drkkz:{mountpoint:/var/lib/kubelet/pods/16e35961-5e37-4e7b-bf8d-5a8d7c273893/volumes/kubernetes.io~projected/kube-api-access-drkkz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/257163a1-1ead-438e-94c2-ca672bdcaa17/volumes/kubernetes.io~projected/kube-api-access-nx298:{mountpoint:/var/lib/kubelet/pods/257163a1-1ead-438e-94c2-ca672bdcaa17/volumes/kubernetes.io~projected/kube-api-access-nx298 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/257163a1-1ead-438e-94c2-ca672bdcaa17/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/257163a1-1ead-438e-94c2-ca672bdcaa17/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33fad09e-a1b4-4998-bb14-8e6e33a3a03c/volumes/kubernetes.io~projected/kube-api-access-jb826:{mountpoint:/var/lib/kubelet/pods/33fad09e-a1b4-4998-bb14-8e6e33a3a03c/volumes/kubernetes.io~projected/kube-api-access-jb826 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~projected/kube-api-access-zq8x5:{mountpoint:/var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~projected/kube-api-access-zq8x5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~projected/kube-api-access-74x4f:{mountpoint:/var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~projected/kube-api-access-74x4f major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5291323c-ea92-499c-b246-a4f692f3f8c5/volumes/kubernetes.io~projected/kube-api-access-8x5g6:{mountpoint:/var/lib/kubelet/pods/5291323c-ea92-499c-b246-a4f692f3f8c5/volumes/kubernetes.io~projected/kube-api-access-8x5g6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volumes/kubernetes.io~projected/kube-api-access-79jtm:{mountpoint:/var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volumes/kubernetes.io~projected/kube-api-access-79jtm major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/790cdff4-1780-4548-a327-e7b7df474347/volumes/kubernetes.io~projected/kube-api-access-4xt65:{mountpoint:/var/lib/kubelet/pods/790cdff4-1780-4548-a327-e7b7df474347/volumes/kubernetes.io~projected/kube-api-access-4xt65 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/814f4474-5a86-4ddf-8bd8-beeebe917d71/volumes/kubernetes.io~projected/kube-api-access-fkp9k:{mountpoint:/var/lib/kubelet/pods/814f4474-5a86-4ddf-8bd8-beeebe917d71/volumes/kubernetes.io~projected/kube-api-access-fkp9k major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88b783e5-08d6-44c6-ac37-b50adf8736ea/volumes/kubernetes.io~projected/kube-api-access-8wsdw:{mountpoint:/var/lib/kubelet/pods/88b783e5-08d6-44c6-ac37-b50adf8736ea/volumes/kubernetes.io~projected/kube-api-access-8wsdw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88b783e5-08d6-44c6-ac37-b50adf8736ea/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/88b783e5-08d6-44c6-ac37-b50adf8736ea/volumes/kubernetes.io~secret/metrics-certs major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~projected/kube-api-access-dg6l9:{mountpoint:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~projected/kube-api-access-dg6l9 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9/volumes/kubernetes.io~projected/kube-api-access-wvzgw:{mountpoint:/var/lib/kubelet/pods/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9/volumes/kubernetes.io~projected/kube-api-access-wvzgw major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a14e8823-8f98-40c4-b20d-dfea320c8304/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a14e8823-8f98-40c4-b20d-dfea320c8304/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b29d373a-4cfd-4d5b-8561-e604264326db/volumes/kubernetes.io~projected/kube-api-access-ml4xv:{mountpoint:/var/lib/kubelet/pods/b29d373a-4cfd-4d5b-8561-e604264326db/volumes/kubernetes.io~projected/kube-api-access-ml4xv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b34d67be-03c6-42d2-8e6f-d5936d7d940b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b34d67be-03c6-42d2-8e6f-d5936d7d940b/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdde073d-a59a-4953-8650-3c997810c4b9/volumes/kubernetes.io~projected/kube-api-access-drrgf:{mountpoint:/var/lib/kubelet/pods/cdde073d-a59a-4953-8650-3c997810c4b9/volumes/kubernetes.io~projected/kube-api-access-drrgf major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdde073d-a59a-4953-8650-3c997810c4b9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/cdde073d-a59a-4953-8650-3c997810c4b9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5655095-e60c-40a9-9f0d-510e96cdd667/volumes/kubernetes.io~projected/kube-api-access-tvmcr:{mountpoint:/var/lib/kubelet/pods/d5655095-e60c-40a9-9f0d-510e96cdd667/volumes/kubernetes.io~projected/kube-api-access-tvmcr major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5655095-e60c-40a9-9f0d-510e96cdd667/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/d5655095-e60c-40a9-9f0d-510e96cdd667/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~projected/kube-api-access-4z6zn:{mountpoint:/var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~projected/kube-api-access-4z6zn major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/10ac8225c47a149cb3633e4ccbc7ef3943be72c6e1bd043081c51e81717fbad4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/93f969e2a6e865b5fc5a69552831a2bd12213c8a83db0dd087cb57d3548729b1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c4ee49d5f4061d53f1dd97a13d368402b8f18260a0b792180e701b00d15320b8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c2fa7936c09999c91ac1d39f06fb8f4b0773db05b64c9906714c46507bc4e083/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9f67d029f2ae750fd3ed8c2413153e9689b89288a70ab99636502870c8ecbbc0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7029c0061f94438da3c09311f11e13b9fdff6ab42f285de2245de4c4dca365cf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/15226124fbe16a07028b6d3884dbf86053d6f95c6fee40547f3b67cfc08e301b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a66027c5595b72c8632b6b5ec253aeb7ee0e5cc454ac153fd2ebfd8c10ab8077/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/eff302f037831cd199c8793a6750948154fcf81b3769702173f182ff188f9da9/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/46d5487e7d2c6865fd32613d299ea01551711645c93fcb45df2f37a825828fa8/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/87efe2df013b7dee8c018ff50b3b293447c2ad33b9c1f596a757329ea4832328/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/7d7b7d53d74cf883a2dc21d031b45ed8ff73f7dc9be1a8728a93fc9044d4528a/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/6acf211950842af041879a93bb13ad321b4f98989c2c26329a70296c28d79353/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/018c756815ff77c0b2ed219e8863e3b2b35f1d78ae0f805e554ea28ed59e183f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/8c7f35a45b1e913efe2b0e05065ad1e6f2fb31a92382c1424583b25722a4ff92/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e20f7debd5dc655669edc46902b758baeddd48856ef20a34df6d00b2d7f7ce3c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/15fbdf4ce5d601ab9572ab73430dd3a7c3cdf331e7051d90ee39442c42c288b5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/362cba12695ad4e03865e034273d006b397b7aec54936efaee97ce391b398964/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8404d103fe039a478659c37600d737505ca43c499f7b6dc787f06a063f9e7a40/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/edf4b448352518fdb8b3535f9e6b63524dbf00baacd85278de024bbd5991e76b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/65600d3271c8d2c4f4af6172b7b3f92f7db5f6c350e2abb468c5f0ef5ada4406/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/549038a03615fc96320172d10a2cfc6ae7f3cbb75385100bca85334a9ddd3683/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/5d0f1d291de6cd0a11873a9ffe47726899df490322ef72668cb1a28081493001/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d4aaede85a0f7a848d0eec70abd163af1a73b92f2c5eea61a609c7061fa84d33/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/4e6fa23ac3801eb675c1a476351769e34ec0d082d0eab7fe443336e786ab9070/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/6792c006a97a72ef90fecaf568d06741ed6badba7978b97f11e394a18440a476/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/fe3f401105d6dd4d3c17567cfadce2d656364b6c8bee19b1774896b3a0a8186e/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/326039b40463f4fa6ae6fa6c2e19c9ffd154385c98e4585bc7f3efac80cae77d/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/6559fd273225de8834114837d6d65729c7320b8b4574c9b16924160b17439586/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/18439cf24ad1057e3ab6fe24918579f700342e2d4480da119ee2311efaed7307/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/89adea6b69f26b3fc5f78efbe98277248343163ff991a0ef8ad5e9727667cbb0/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/c8aa1655176cba285e398c94dc3de7a78b7dd5820354a20c3d5d47e634380dcd/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/f4394c209c3fdaaf8fac3a49867767b94c29a1b6f9bd231fd88ca47e61e40ec3/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/e1e77cef9bafd915d788a0a284827f2e90bf0d742efcba8b50e6abd891508837/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/9785fb76eec994b85c437bbe1b53ec7e023168d7a893279a56377e26d53e7734/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/6ce0b5026b05bf849a63949b2c84f38c197c3fe039495ca1395188fe81cb057f/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/28d41c5c6f031e706802ab9511dc8c06b81716a4abd98a4a5f7f91ba49bd0648/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/2a31522919c399773395a3b00bd98069727da7cae5abda10b7d21af864ceed77/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/ab76ff16f2b82e125efe7dc6b37716dccb8739a777fe60ae532b24c720ef4070/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/a39334b422a34a80ded79bbd0e9da26e3eefefb885b3ccbd18c8ab3dec41e887/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/8ca677878148f4b926b0310670a36479e37de66155e17d0e9791a017ab2df90b/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/04c97416780eb94900dac284af29a8070530de56e023327d8966f31df1d67d7b/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/0b1ae260d9fbc77c51adb25ad4615b922ea5f6bf3d7a8042c6f0646de448f75e/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/57e9a7244e4ef52c8e07c367d2d538e8c6ac416ef187113c02c39b2843c2343b/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/d594055fbf413f322d6a25b2443992f146eea123a97f83ca9bee1756d161ac28/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/3dc29ea77969ee6c607678643689977d734c628623ffd4be2eddefe2f9a3bd49/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/dde76336f3fe1ffc6d7c3c8b99e81b19883a0e92307f3118aeecdd0a7edbee83/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/7be102853ab7e44512378af68788cfc8cca18f39bf51abc061e69fa1a7da39d0/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/cf19117f5513d1625b3aa767bc530862223370adb58167c62c61670762d1a6cf/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/d44184dcec0e1cb76b5d2fffbeab5e76a2b6ca638e19526679d14d323e0a51bd/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/1e667347a8539bfc51a7f401205e4ce284320536bdd894b5c54f4ea6a7ac5fb9/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/9f04beca2be6ca55d0aa597ad8eda200dfe01c955d452f2daedf78aa103b879a/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/50a14145ca868a9bd53ca3021db54d1266be352036caae274c122180f5308408/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d4a65e536558515497e9cc59ac6897442c3bc3fad1ccd33b922e68066ea0ec28/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5a29a87330f0da4b5e27fdad482ac5097ee799ad048ab7b318e8768b171fae4a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/9ea16f7d7a22e01faf2afa426fb2884a93fa5ed74c1eb7bf6e6caa5dad76a820/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4b735cd19e259e3b00a963114966e733b69ab97c1675acaa8e6778e0248a4145/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/8f077160fc6e081d261f6de593b613f7267cd2da0ec749e95b1b09d63adfaf1b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6f917b7ffb7316c9c2bb21fce487b807ecddc2f42cabffe9526379cb5794fc90/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/4dcee9703b840462472518dfb3f21fd4e15b1cd6361a75b36be6955b68e438c5/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/fadf35f2179782b62a9480ed26abd1da9e8af30bea416f67404c31c7746d8a71/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b74aebfe3f7c672b0b6d694af69700a5a5848de20cbb42ca3f98371044a08eb1/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.351876 5591 manager.go:217] Machine: {Timestamp:2026-11-02 13:44:11.351395243 +0000 UTC m=+0.077017005 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099893 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec222168b52a39c866fc9f10df963976 SystemUUID:ec222168-b52a-39c8-66fc-9f10df963976 BootID:8672f832-69ed-4b4b-96ce-dec1f4607134 Filesystems:[{Device:/var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volumes/kubernetes.io~projected/kube-api-access-79jtm DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88b783e5-08d6-44c6-ac37-b50adf8736ea/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~projected/kube-api-access-4z6zn DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/790cdff4-1780-4548-a327-e7b7df474347/volumes/kubernetes.io~projected/kube-api-access-4xt65 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd9f1c9b783cdf5120ff0e063a238cc5dc52425e2f6ddeba4b5d77eabad36e80/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccb8ebaf7df6a5841a53771b5ebfd0caf034c96f71b441bb9d37a1605e28906c/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0b381cdaaa15073f9edc1b62f135749f109532ba4839ac2d7fcdb6dffc0eabc/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5655095-e60c-40a9-9f0d-510e96cdd667/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~projected/kube-api-access-dg6l9 DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b34d67be-03c6-42d2-8e6f-d5936d7d940b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~projected/kube-api-access-zq8x5 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8754eb72fb1713029ecad4fd8fb52c85dc14c5ee40aab2c9498793eb2a2e1f33/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7688829976fff75a5c9900b8ae22fb29699368757ec7d6cba1610fb787f5f03/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88b783e5-08d6-44c6-ac37-b50adf8736ea/volumes/kubernetes.io~projected/kube-api-access-8wsdw DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a14e8823-8f98-40c4-b20d-dfea320c8304/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d7ca799ea3892c4ea1b4495d403cc484b6e5f5d05d7f529f286b04698d77783/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33fad09e-a1b4-4998-bb14-8e6e33a3a03c/volumes/kubernetes.io~projected/kube-api-access-jb826 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~projected/kube-api-access-74x4f DeviceMajor:0 DeviceMinor:243 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71fbdaae8e4207f512c28ea54b6db30c72bc3a0c775210fba12a1f0586e655ba/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/257163a1-1ead-438e-94c2-ca672bdcaa17/volumes/kubernetes.io~projected/kube-api-access-nx298 DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9aa2152568f8536cf170e981fa4df2945208c1d84af2097b5e997667d4422530/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12ba0dc43d43f4899ac81c7c60b103ec4137895ff6c963087c531e717d539d56/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80565425f70ad633a65bab7ce1d5af11122c5830bb78de1737297d3b4a7a8dcd/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/782560dab0bfeb115690883ff55d5420c3f616779496c2516ee761093748e55c/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20721192a61ad5faf3b6c895c5797a090afb06049309a20d9a5cc67939b4a3e0/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdde073d-a59a-4953-8650-3c997810c4b9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/257163a1-1ead-438e-94c2-ca672bdcaa17/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16e35961-5e37-4e7b-bf8d-5a8d7c273893/volumes/kubernetes.io~projected/kube-api-access-drkkz DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6caba99e97540d491261dd7a170d10ff675ac0a6ccc14a1feef24fb6e9600bdb/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5655095-e60c-40a9-9f0d-510e96cdd667/volumes/kubernetes.io~projected/kube-api-access-tvmcr DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b4dbff4c5dfe453899913d18841cc6cce1d59da441d1b4c45b3767e487636a8/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e273bc-9f51-4993-86c2-7e02c3b88c7d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2e469850889f5454ef32b7c2a7741f5a31573051185b3e7a5e8778b2874f7f4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98a7d07b-33d6-4778-8bb0-7c277468d407/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9/volumes/kubernetes.io~projected/kube-api-access-wvzgw DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0ef62365-5210-41a9-a45c-7e71cd04fb4c/volumes/kubernetes.io~projected/kube-api-access-9z52s DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c49d4d28ce99735571e297417189ae9fca94ef8dd4923c7c6390df1f0daad08/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/814f4474-5a86-4ddf-8bd8-beeebe917d71/volumes/kubernetes.io~projected/kube-api-access-fkp9k DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e0d4d8a8a7c4fa08c9e19f47c0a644c400e19336a7a83c887d29deeebebaf4d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c484b97bf8b5f0634d95075477c05abe6a73beb822241db51b0421b2152e197/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ef62365-5210-41a9-a45c-7e71cd04fb4c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cdde073d-a59a-4953-8650-3c997810c4b9/volumes/kubernetes.io~projected/kube-api-access-drrgf DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b29d373a-4cfd-4d5b-8561-e604264326db/volumes/kubernetes.io~projected/kube-api-access-ml4xv DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5291323c-ea92-499c-b246-a4f692f3f8c5/volumes/kubernetes.io~projected/kube-api-access-8x5g6 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b17ed04db612c0c3334014fe1a12751eadf16567f191474709612605d65f0351/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ba2e9dcaade4fd7469acc78b70fb7d03a3a3138d118f3d16b033720ea23d62c/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36315738-ca55-4bbb-8644-748e4ae00efe/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b4dbff4c5dfe45 MacAddress:52:84:80:d8:cb:7f Speed:10000 Mtu:8901} {Name:0c49d4d28ce9973 MacAddress:ae:22:f3:2d:00:66 Speed:10000 Mtu:8901} {Name:20721192a61ad5f MacAddress:e6:58:98:74:3b:51 Speed:10000 Mtu:8901} {Name:8754eb72fb17130 MacAddress:3e:40:a1:8e:c2:2c Speed:10000 Mtu:8901} {Name:8ba2e9dcaade4fd MacAddress:1a:03:cd:13:3e:b0 Speed:10000 Mtu:8901} {Name:9aa2152568f8536 MacAddress:46:3a:06:ee:83:97 Speed:10000 Mtu:8901} {Name:b17ed04db612c0c MacAddress:42:ea:9b:d9:c2:c9 Speed:10000 Mtu:8901} {Name:bd9f1c9b783cdf5 MacAddress:66:bc:24:05:c9:61 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:db:bb:0f:0d Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:11:c7:57:ca:fa Speed:0 Mtu:8901} {Name:c0b381cdaaa1507 MacAddress:de:b6:ba:59:a4:a3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:db:bb:0f:0d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:6e:63:4f:9d:e1:19 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:68:a9:a1:c0:bb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.352254 5591 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.352300 5591 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.352717 5591 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.352740 5591 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.352876 5591 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.352884 5591 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.352898 5591 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.352922 5591 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.353038 5591 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.353220 5591 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.353326 5591 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.353358 5591 kubelet.go:493] "Attempting to sync node with API server" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.353371 5591 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.353380 5591 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.353411 5591 kubelet.go:405] "Adding apiserver pod source" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.353420 5591 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.355213 5591 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.355227 5591 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.357043 5591 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362138 5591 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362155 5591 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362322 5591 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362582 5591 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362624 5591 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362734 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362749 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362762 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362771 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362781 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362790 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362800 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362808 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362859 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362870 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362883 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362897 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362928 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 13:44:11.363718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.362937 5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 13:44:11.372032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.367282 5591 server.go:1263] "Started kubelet" Nov 02 13:44:11.372032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.367811 5591 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 13:44:11.372032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.368840 5591 server.go:317] "Adding debug handlers to kubelet server" Nov 02 13:44:11.372032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.369067 5591 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 13:44:11.372032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.369184 5591 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 13:44:11.372032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.369615 5591 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 13:44:11.367867 ip-10-0-140-236 systemd[1]: Started Kubernetes Kubelet. Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.372059 5591 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.372191 5591 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.372444 5591 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:38:01 +0000 UTC" deadline="2028-08-13 21:53:45.647356086 +0000 UTC" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.372482 5591 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15608h9m34.274877622s" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.372669 5591 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.372679 5591 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.372783 5591 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375379 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36315738-ca55-4bbb-8644-748e4ae00efe" volumeName="kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-etc-tuned" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375431 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e273bc-9f51-4993-86c2-7e02c3b88c7d" volumeName="kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-metrics-client-ca" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375448 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ef62365-5210-41a9-a45c-7e71cd04fb4c" volumeName="kubernetes.io/secret/0ef62365-5210-41a9-a45c-7e71cd04fb4c-cert" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375471 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52eda2b9-8258-4f7e-bc72-6ad9f6f1167a" volumeName="kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-env-overrides" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375484 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36315738-ca55-4bbb-8644-748e4ae00efe" volumeName="kubernetes.io/projected/36315738-ca55-4bbb-8644-748e4ae00efe-kube-api-access-zq8x5" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375513 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375533 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdde073d-a59a-4953-8650-3c997810c4b9" volumeName="kubernetes.io/secret/cdde073d-a59a-4953-8650-3c997810c4b9-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375546 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52eda2b9-8258-4f7e-bc72-6ad9f6f1167a" volumeName="kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-config" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375562 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/configmap/98a7d07b-33d6-4778-8bb0-7c277468d407-metrics-client-ca" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375588 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/empty-dir/98a7d07b-33d6-4778-8bb0-7c277468d407-tmp" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375602 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/projected/98a7d07b-33d6-4778-8bb0-7c277468d407-kube-api-access-dg6l9" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375614 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b29d373a-4cfd-4d5b-8561-e604264326db" volumeName="kubernetes.io/projected/b29d373a-4cfd-4d5b-8561-e604264326db-kube-api-access-ml4xv" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375634 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5291323c-ea92-499c-b246-a4f692f3f8c5" volumeName="kubernetes.io/configmap/5291323c-ea92-499c-b246-a4f692f3f8c5-serviceca" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375654 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="814f4474-5a86-4ddf-8bd8-beeebe917d71" volumeName="kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-daemon-config" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375681 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e35961-5e37-4e7b-bf8d-5a8d7c273893" volumeName="kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375703 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e35961-5e37-4e7b-bf8d-5a8d7c273893" volumeName="kubernetes.io/projected/16e35961-5e37-4e7b-bf8d-5a8d7c273893-kube-api-access-drkkz" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375717 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="257163a1-1ead-438e-94c2-ca672bdcaa17" volumeName="kubernetes.io/empty-dir/257163a1-1ead-438e-94c2-ca672bdcaa17-tmp-dir" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375739 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790cdff4-1780-4548-a327-e7b7df474347" volumeName="kubernetes.io/projected/790cdff4-1780-4548-a327-e7b7df474347-kube-api-access-4xt65" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375760 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790cdff4-1780-4548-a327-e7b7df474347" volumeName="kubernetes.io/empty-dir/790cdff4-1780-4548-a327-e7b7df474347-tmp-dir" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375782 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="814f4474-5a86-4ddf-8bd8-beeebe917d71" volumeName="kubernetes.io/projected/814f4474-5a86-4ddf-8bd8-beeebe917d71-kube-api-access-fkp9k" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375794 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5655095-e60c-40a9-9f0d-510e96cdd667" volumeName="kubernetes.io/projected/d5655095-e60c-40a9-9f0d-510e96cdd667-kube-api-access-tvmcr" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375828 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e273bc-9f51-4993-86c2-7e02c3b88c7d" volumeName="kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375842 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="257163a1-1ead-438e-94c2-ca672bdcaa17" volumeName="kubernetes.io/configmap/257163a1-1ead-438e-94c2-ca672bdcaa17-config-volume" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375862 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35" volumeName="kubernetes.io/projected/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-kube-api-access-74x4f" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375894 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-grpc-tls" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375914 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375926 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34d67be-03c6-42d2-8e6f-d5936d7d940b" volumeName="kubernetes.io/secret/b34d67be-03c6-42d2-8e6f-d5936d7d940b-agent-certs" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375947 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdde073d-a59a-4953-8650-3c997810c4b9" volumeName="kubernetes.io/configmap/cdde073d-a59a-4953-8650-3c997810c4b9-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375965 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5291323c-ea92-499c-b246-a4f692f3f8c5" volumeName="kubernetes.io/projected/5291323c-ea92-499c-b246-a4f692f3f8c5-kube-api-access-8x5g6" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.375978 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88b783e5-08d6-44c6-ac37-b50adf8736ea" volumeName="kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376015 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b29d373a-4cfd-4d5b-8561-e604264326db" volumeName="kubernetes.io/configmap/b29d373a-4cfd-4d5b-8561-e604264326db-iptables-alerter-script" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376042 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e35961-5e37-4e7b-bf8d-5a8d7c273893" volumeName="kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-binary-copy" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376056 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e35961-5e37-4e7b-bf8d-5a8d7c273893" volumeName="kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376097 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52eda2b9-8258-4f7e-bc72-6ad9f6f1167a" volumeName="kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-script-lib" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376111 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb24650-63d2-4d50-9a20-db9bf4a5e2c9" volumeName="kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376123 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e273bc-9f51-4993-86c2-7e02c3b88c7d" volumeName="kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-tls" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376144 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="257163a1-1ead-438e-94c2-ca672bdcaa17" volumeName="kubernetes.io/secret/257163a1-1ead-438e-94c2-ca672bdcaa17-metrics-tls" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376156 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33fad09e-a1b4-4998-bb14-8e6e33a3a03c" volumeName="kubernetes.io/projected/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kube-api-access-jb826" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376168 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52eda2b9-8258-4f7e-bc72-6ad9f6f1167a" volumeName="kubernetes.io/secret/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376191 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35" volumeName="kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-tls" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376206 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52eda2b9-8258-4f7e-bc72-6ad9f6f1167a" volumeName="kubernetes.io/projected/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-kube-api-access-79jtm" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376228 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376265 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e273bc-9f51-4993-86c2-7e02c3b88c7d" volumeName="kubernetes.io/projected/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-kube-api-access-4z6zn" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376278 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="257163a1-1ead-438e-94c2-ca672bdcaa17" volumeName="kubernetes.io/projected/257163a1-1ead-438e-94c2-ca672bdcaa17-kube-api-access-nx298" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376304 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36315738-ca55-4bbb-8644-748e4ae00efe" volumeName="kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-tmp" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376331 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376344 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14e8823-8f98-40c4-b20d-dfea320c8304" volumeName="kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376358 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdde073d-a59a-4953-8650-3c997810c4b9" volumeName="kubernetes.io/projected/cdde073d-a59a-4953-8650-3c997810c4b9-kube-api-access-drrgf" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376377 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e273bc-9f51-4993-86c2-7e02c3b88c7d" volumeName="kubernetes.io/empty-dir/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-textfile" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.376397 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e273bc-9f51-4993-86c2-7e02c3b88c7d" volumeName="kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:11.385771 5591 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.389517 5591 factory.go:55] Registering systemd factory Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.390994 5591 factory.go:223] Registration of the systemd container factory successfully Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.390922 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ef62365-5210-41a9-a45c-7e71cd04fb4c" volumeName="kubernetes.io/projected/0ef62365-5210-41a9-a45c-7e71cd04fb4c-kube-api-access-9z52s" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391305 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35" volumeName="kubernetes.io/configmap/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-metrics-client-ca" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391335 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35" volumeName="kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391354 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88b783e5-08d6-44c6-ac37-b50adf8736ea" volumeName="kubernetes.io/projected/88b783e5-08d6-44c6-ac37-b50adf8736ea-kube-api-access-8wsdw" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391379 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a7d07b-33d6-4778-8bb0-7c277468d407" volumeName="kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391395 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34d67be-03c6-42d2-8e6f-d5936d7d940b" volumeName="kubernetes.io/configmap/b34d67be-03c6-42d2-8e6f-d5936d7d940b-konnectivity-ca" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391446 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdde073d-a59a-4953-8650-3c997810c4b9" volumeName="kubernetes.io/empty-dir/cdde073d-a59a-4953-8650-3c997810c4b9-data-volume" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391465 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="814f4474-5a86-4ddf-8bd8-beeebe917d71" volumeName="kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-cni-binary-copy" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391487 5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5655095-e60c-40a9-9f0d-510e96cdd667" volumeName="kubernetes.io/secret/d5655095-e60c-40a9-9f0d-510e96cdd667-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391501 5591 reconstruct.go:97] "Volume reconstruction finished" Nov 02 13:44:11.397266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.391510 5591 reconciler.go:29] "Reconciler: start to sync state" Nov 02 13:44:11.400612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.397792 5591 factory.go:153] Registering CRI-O factory Nov 02 13:44:11.400612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.397809 5591 factory.go:223] Registration of the crio container factory successfully Nov 02 13:44:11.400612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.397895 5591 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 13:44:11.400612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.397916 5591 factory.go:103] Registering Raw factory Nov 02 13:44:11.400612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.397933 5591 manager.go:1199] Started watching for new ooms in manager Nov 02 13:44:11.400612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.398290 5591 manager.go:319] Starting recovery of all containers Nov 02 13:44:11.415788 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.415749 5591 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 13:44:11.421863 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.421782 5591 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 13:44:11.421863 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.421817 5591 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 13:44:11.421863 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.421842 5591 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 13:44:11.422165 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:11.421874 5591 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 13:44:11.428022 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.427961 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:44:11.433538 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.433516 5591 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 13:44:11.434098 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.434063 5591 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:44:11.434260 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.434166 5591 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:44:11.435514 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.435492 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:44:11.436945 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.436897 5591 generic.go:356] "Generic (PLEG): container finished" podID="d8e273bc-9f51-4993-86c2-7e02c3b88c7d" containerID="082261beb898c16786edd6810817530e6e3515c5ad03856fe181213860ab2deb" exitCode=0 Nov 02 13:44:11.458606 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.458543 5591 generic.go:356] "Generic (PLEG): container finished" podID="60bef5910d750df47fec5174fe909f02" containerID="48fff5c9b870ae9ae4f525158b5653b2b3a4bd9a27eae95dae9a14941aec6e9e" exitCode=0 Nov 02 13:44:11.467577 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.467532 5591 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:44:11.467916 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.467891 5591 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:44:11.484398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.484369 5591 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="5d8b71e54bd055918f92701a1152fe4ad196f0e56dd4c54c41dbae2e877685cd" exitCode=0 Nov 02 13:44:11.484398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.484394 5591 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="de5c5740c87a5894570a178971801fccccd1ff4a42593758c3b5aab3ab28a523" exitCode=0 Nov 02 13:44:11.484398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.484400 5591 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="e6c446d927ceaf665d1f089efa86c38c99a5d9e1993a23001c7c5a37474e09ab" exitCode=0 Nov 02 13:44:11.484398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.484406 5591 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="555e0dd5e4151b7404d4d76f5c8fd4e41aa6bc56e2fc2b108758e9b11f8bcf1b" exitCode=0 Nov 02 13:44:11.484659 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.484411 5591 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="2079f19251417a9a2e283191b94da176f0fd5bdf980ab5448eac9ddab363689a" exitCode=0 Nov 02 13:44:11.484659 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.484416 5591 generic.go:356] "Generic (PLEG): container finished" podID="16e35961-5e37-4e7b-bf8d-5a8d7c273893" containerID="677f5e07c18fd4f100458255948a8ae11262536d272f7e22d5007ef505871f29" exitCode=0 Nov 02 13:44:11.520475 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.520445 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:44:11.520884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.520854 5591 generic.go:356] "Generic (PLEG): container finished" podID="52eda2b9-8258-4f7e-bc72-6ad9f6f1167a" containerID="d7a60ba7d2f5a5d78d0d248c98a0eeb8325a9f1105b2b849c6e46dc0d44fd514" exitCode=1 Nov 02 13:44:11.521957 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:11.521927 5591 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 13:44:11.531349 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.531326 5591 manager.go:324] Recovery completed Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552494 5591 cpu_manager.go:226] "Starting" policy="none" Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552519 5591 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552550 5591 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552751 5591 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552766 5591 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552794 5591 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552802 5591 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552810 5591 policy_none.go:50] "Start" Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552828 5591 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 13:44:11.552884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.552841 5591 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 13:44:11.555258 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.555222 5591 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 13:44:11.555461 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.555446 5591 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 13:44:11.555514 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.555462 5591 policy_none.go:44] "Start" Nov 02 13:44:11.560154 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.560112 5591 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 13:44:11.564627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.564609 5591 manager.go:341] "Starting Device Plugin manager" Nov 02 13:44:11.565095 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:11.564639 5591 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 13:44:11.565095 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.565094 5591 server.go:87] "Starting device plugin registration server" Nov 02 13:44:11.565358 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.565345 5591 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 13:44:11.565408 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.565357 5591 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 13:44:11.565456 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.565433 5591 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 13:44:11.565559 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.565526 5591 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 13:44:11.565661 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.565562 5591 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 13:44:11.565661 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.565569 5591 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 13:44:11.565661 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.565611 5591 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:44:11.565550014Z","UUID":"4bed9697-6c70-4ae6-a4a7-7b7392f41a23","Handler":null,"Name":"","Endpoint":""} Nov 02 13:44:11.567127 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:11.567105 5591 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 13:44:11.567480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.567458 5591 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 13:44:11.567595 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.567487 5591 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 13:44:11.665906 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.665862 5591 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:44:11.668255 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.668229 5591 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:44:11.668374 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.668273 5591 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:44:11.668374 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.668288 5591 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:44:11.668374 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.668337 5591 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.704234 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.704154 5591 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.704365 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.704253 5591 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.722248 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.722197 5591 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"] Nov 02 13:44:11.722816 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.722792 5591 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.722816 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.722781 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerStarted","Data":"f1f7146ba8df40a865cc2ab7e0239327d6d65ab90acb88a903999aee9344cf81"} Nov 02 13:44:11.723020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.722841 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerDied","Data":"48fff5c9b870ae9ae4f525158b5653b2b3a4bd9a27eae95dae9a14941aec6e9e"} Nov 02 13:44:11.723020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.722848 5591 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.723020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.722857 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerStarted","Data":"1d7ca799ea3892c4ea1b4495d403cc484b6e5f5d05d7f529f286b04698d77783"} Nov 02 13:44:11.723020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.722918 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" event={"ID":"e82d0dbde7394ec25631244e1f1d5cd9","Type":"ContainerStarted","Data":"10b90b2a1003f3a3f7d1a8cf25e4c1071f7239147f0113051e31f42e73405120"} Nov 02 13:44:11.723020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.722931 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" event={"ID":"e82d0dbde7394ec25631244e1f1d5cd9","Type":"ContainerStarted","Data":"71fbdaae8e4207f512c28ea54b6db30c72bc3a0c775210fba12a1f0586e655ba"} Nov 02 13:44:11.765242 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.765198 5591 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:44:11.765415 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:11.765288 5591 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.765415 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.765213 5591 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:44:11.765415 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:11.765390 5591 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.835635 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.835573 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.835635 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.835640 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.835857 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.835688 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.936072 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.936033 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.936072 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.936074 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.936284 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.936098 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.936284 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.936144 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.936284 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.936162 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:11.936284 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:11.936144 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:44:12.354389 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.354354 5591 apiserver.go:52] "Watching apiserver" Nov 02 13:44:12.394574 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.394544 5591 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 13:44:12.395103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.395082 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal","openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4","openshift-monitoring/thanos-querier-c6896987-j2hml","openshift-dns/dns-default-6bll8","openshift-insights/insights-runtime-extractor-87kgf","openshift-network-diagnostics/network-check-target-cc5c8","openshift-ingress-canary/ingress-canary-j79zx","kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal","openshift-multus/multus-additional-cni-plugins-vbv78","openshift-dns/node-resolver-j9lm6","openshift-ovn-kubernetes/ovnkube-node-mq8vn","openshift-multus/multus-g268n","openshift-monitoring/node-exporter-sjj7t","openshift-cluster-node-tuning-operator/tuned-ttzxz","kube-system/global-pull-secret-syncer-scx6x","openshift-multus/network-metrics-daemon-v8n7n","openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p","openshift-image-registry/node-ca-9d7n9","openshift-network-operator/iptables-alerter-ff8sv","kube-system/konnectivity-agent-8hhsr"] Nov 02 13:44:12.412493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.412455 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 13:44:12.412493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.412481 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 13:44:12.412683 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.412510 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.412683 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.412542 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bqvhh\"" Nov 02 13:44:12.412683 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.412662 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 13:44:12.412851 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.412835 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 13:44:12.412900 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.412861 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 13:44:12.413132 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413116 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qnz8k\"" Nov 02 13:44:12.413200 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413140 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 13:44:12.413324 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413291 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.413439 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413419 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-47zgd\"" Nov 02 13:44:12.413492 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413309 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-52vg6\"" Nov 02 13:44:12.413550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413489 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.413600 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413570 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 13:44:12.413651 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413596 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.413710 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413686 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 13:44:12.413950 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.413921 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.414129 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.414074 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6lw86\"" Nov 02 13:44:12.415576 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.415557 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.422763 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.422743 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 13:44:12.422926 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.422911 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5dqzs\"" Nov 02 13:44:12.423228 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.423214 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w4l5x\"" Nov 02 13:44:12.423369 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.423355 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.423453 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.423438 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lj29d\"" Nov 02 13:44:12.423491 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.423475 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.423597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.423586 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fs4v4\"" Nov 02 13:44:12.423681 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.423670 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.423730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.423699 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.423790 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.423776 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4wcws\"" Nov 02 13:44:12.426214 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.426198 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.426716 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.426701 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.426716 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.426712 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.427152 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427127 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.427199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427174 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 13:44:12.427199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427174 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 13:44:12.427276 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427262 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.427610 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427599 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 13:44:12.427641 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427635 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 13:44:12.427682 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427665 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 13:44:12.427772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427759 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lvdl4\"" Nov 02 13:44:12.427835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427812 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.427835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427815 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hkz22\"" Nov 02 13:44:12.427934 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427864 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 13:44:12.427934 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427917 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.428021 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.427958 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.428130 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428115 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 13:44:12.428184 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428129 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.428184 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428135 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 13:44:12.428257 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428192 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.428257 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428211 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 13:44:12.428257 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428211 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 13:44:12.428397 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428312 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 13:44:12.428461 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428447 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 13:44:12.428510 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428482 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 13:44:12.428510 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428495 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 13:44:12.428572 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428512 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 13:44:12.428616 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428601 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xwmgj\"" Nov 02 13:44:12.428661 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428605 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 13:44:12.428661 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428636 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 13:44:12.428661 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428607 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 13:44:12.428797 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.428723 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 13:44:12.429808 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429782 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 13:44:12.429808 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429788 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 13:44:12.429808 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429798 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cv5n8\"" Nov 02 13:44:12.429952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429810 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tg4rm\"" Nov 02 13:44:12.429952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429810 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-16km2qrs03voi\"" Nov 02 13:44:12.429952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429828 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 13:44:12.429952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429831 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 13:44:12.429952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429845 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 13:44:12.429952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429848 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 13:44:12.429952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429849 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hnbmg\"" Nov 02 13:44:12.429952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.429888 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fpdql\"" Nov 02 13:44:12.435398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.435383 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wff6\"" Nov 02 13:44:12.435492 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.435451 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9sg29\"" Nov 02 13:44:12.438157 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.438131 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:12.474689 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.474660 5591 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 13:44:12.527654 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.527628 5591 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:44:12.527784 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.527708 5591 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:12.538679 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538657 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/257163a1-1ead-438e-94c2-ca672bdcaa17-metrics-tls\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:12.538723 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538687 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-device-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.538723 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538705 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.538723 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538721 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-kubelet\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.538830 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538767 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6l9\" (UniqueName: \"kubernetes.io/projected/98a7d07b-33d6-4778-8bb0-7c277468d407-kube-api-access-dg6l9\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.538830 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538785 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-netns\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.538830 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538801 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.538924 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538869 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-var-lib-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.538924 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538907 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovn-node-metrics-cert\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.539008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.538953 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-etc-kubernetes\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.539050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539034 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/257163a1-1ead-438e-94c2-ca672bdcaa17-metrics-tls\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:12.539089 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539041 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-tls\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.539212 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539193 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovn-node-metrics-cert\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.539251 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539211 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-tls\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.539251 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539198 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xt65\" (UniqueName: \"kubernetes.io/projected/790cdff4-1780-4548-a327-e7b7df474347-kube-api-access-4xt65\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:44:12.539325 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539316 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b29d373a-4cfd-4d5b-8561-e604264326db-host-slash\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:44:12.539356 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539344 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx298\" (UniqueName: \"kubernetes.io/projected/257163a1-1ead-438e-94c2-ca672bdcaa17-kube-api-access-nx298\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:12.539390 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539362 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:44:12.539433 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539403 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.539514 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539502 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98a7d07b-33d6-4778-8bb0-7c277468d407-metrics-client-ca\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.539552 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539525 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.539587 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539560 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-netns\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.539617 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539581 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:44:12.539676 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539659 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b34d67be-03c6-42d2-8e6f-d5936d7d940b-agent-certs\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:44:12.539730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539711 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.539763 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539727 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98a7d07b-33d6-4778-8bb0-7c277468d407-metrics-client-ca\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.539763 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539695 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5655095-e60c-40a9-9f0d-510e96cdd667-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:12.539825 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539765 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.539825 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539784 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-modprobe-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.539825 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539721 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.539825 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539802 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cnibin\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.539942 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539893 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-conf-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.539942 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539927 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5655095-e60c-40a9-9f0d-510e96cdd667-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:12.540047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539938 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.540047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539947 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z52s\" (UniqueName: \"kubernetes.io/projected/0ef62365-5210-41a9-a45c-7e71cd04fb4c-kube-api-access-9z52s\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:44:12.540047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539951 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b34d67be-03c6-42d2-8e6f-d5936d7d940b-agent-certs\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:44:12.540047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.539975 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-run\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.540186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540053 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-tmp\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.540186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540077 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-sys-fs\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.540186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540107 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-metrics-client-ca\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.540186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540121 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-tmp\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.540186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540132 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-script-lib\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.540186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540157 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4xv\" (UniqueName: \"kubernetes.io/projected/b29d373a-4cfd-4d5b-8561-e604264326db-kube-api-access-ml4xv\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:44:12.540186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540182 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/5291323c-ea92-499c-b246-a4f692f3f8c5-kube-api-access-8x5g6\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540206 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-node-log\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540221 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/790cdff4-1780-4548-a327-e7b7df474347-hosts-file\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540244 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540295 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-metrics-client-ca\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540309 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-registration-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540356 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-script-lib\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540395 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540428 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-kubelet-config\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540444 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-dbus\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540464 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cdde073d-a59a-4953-8650-3c997810c4b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.540493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540496 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-wtmp\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540514 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-config\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540551 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-cni-binary-copy\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540586 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790cdff4-1780-4548-a327-e7b7df474347-tmp-dir\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540593 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b783e5-08d6-44c6-ac37-b50adf8736ea-metrics-certs\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540611 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98a7d07b-33d6-4778-8bb0-7c277468d407-tmp\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540643 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-var-lib-kubelet\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540651 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790cdff4-1780-4548-a327-e7b7df474347-tmp-dir\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540672 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5291323c-ea92-499c-b246-a4f692f3f8c5-host\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540677 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-ovnkube-config\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540682 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cdde073d-a59a-4953-8650-3c997810c4b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540721 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98a7d07b-33d6-4778-8bb0-7c277468d407-tmp\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540725 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdde073d-a59a-4953-8650-3c997810c4b9-data-volume\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540764 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540795 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef62365-5210-41a9-a45c-7e71cd04fb4c-cert\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540792 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-cni-binary-copy\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540821 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257163a1-1ead-438e-94c2-ca672bdcaa17-tmp-dir\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540829 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdde073d-a59a-4953-8650-3c997810c4b9-data-volume\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540910 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsdw\" (UniqueName: \"kubernetes.io/projected/88b783e5-08d6-44c6-ac37-b50adf8736ea-kube-api-access-8wsdw\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:44:12.540952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540927 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-socket-dir-parent\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.540974 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257163a1-1ead-438e-94c2-ca672bdcaa17-tmp-dir\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541024 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cdde073d-a59a-4953-8650-3c997810c4b9-crio-socket\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541059 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541026 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef62365-5210-41a9-a45c-7e71cd04fb4c-cert\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541085 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-ovn\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541136 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x4f\" (UniqueName: \"kubernetes.io/projected/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-kube-api-access-74x4f\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541157 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-system-cni-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541175 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541192 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541191 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b34d67be-03c6-42d2-8e6f-d5936d7d940b-konnectivity-ca\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541234 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-netd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541289 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541354 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5291323c-ea92-499c-b246-a4f692f3f8c5-serviceca\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541380 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-etc-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541397 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8x5\" (UniqueName: \"kubernetes.io/projected/36315738-ca55-4bbb-8644-748e4ae00efe-kube-api-access-zq8x5\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541407 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541414 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b34d67be-03c6-42d2-8e6f-d5936d7d940b-konnectivity-ca\") pod \"konnectivity-agent-8hhsr\" (UID: \"b34d67be-03c6-42d2-8e6f-d5936d7d940b\") " pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541414 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541450 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541514 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-root\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541608 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-systemd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541635 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-systemd-units\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541665 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541690 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-kubernetes\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541692 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541718 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6zn\" (UniqueName: \"kubernetes.io/projected/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-kube-api-access-4z6zn\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.541739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541741 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmcr\" (UniqueName: \"kubernetes.io/projected/d5655095-e60c-40a9-9f0d-510e96cdd667-kube-api-access-tvmcr\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541791 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541825 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b29d373a-4cfd-4d5b-8561-e604264326db-iptables-alerter-script\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541860 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5291323c-ea92-499c-b246-a4f692f3f8c5-serviceca\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.541905 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-multus\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542013 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-hostroot\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542035 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-daemon-config\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542013 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542039 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b29d373a-4cfd-4d5b-8561-e604264326db-iptables-alerter-script\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542051 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-textfile\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542084 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542111 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-grpc-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542134 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/257163a1-1ead-438e-94c2-ca672bdcaa17-config-volume\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542138 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-textfile\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542159 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-sys\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542181 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-daemon-config\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542263 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-kubelet\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542300 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/257163a1-1ead-438e-94c2-ca672bdcaa17-config-volume\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542302 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jtm\" (UniqueName: \"kubernetes.io/projected/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-kube-api-access-79jtm\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542315 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542359 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-grpc-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542358 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542430 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb826\" (UniqueName: \"kubernetes.io/projected/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kube-api-access-jb826\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542467 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-multus-certs\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542509 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-host\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542532 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542548 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542580 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-bin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542613 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkp9k\" (UniqueName: \"kubernetes.io/projected/814f4474-5a86-4ddf-8bd8-beeebe917d71-kube-api-access-fkp9k\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542633 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysconfig\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542675 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-lib-modules\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542703 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-etc-tuned\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542719 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-cnibin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542741 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-log-socket\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542786 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36315738-ca55-4bbb-8644-748e4ae00efe-etc-tuned\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.542795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542786 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542887 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542917 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542919 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrgf\" (UniqueName: \"kubernetes.io/projected/cdde073d-a59a-4953-8650-3c997810c4b9-kube-api-access-drrgf\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.542949 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543019 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-conf\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543035 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-systemd\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543062 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-system-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543086 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cdde073d-a59a-4953-8650-3c997810c4b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543129 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543175 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-env-overrides\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543204 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-sys\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543215 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cdde073d-a59a-4953-8650-3c997810c4b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543226 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543243 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkkz\" (UniqueName: \"kubernetes.io/projected/16e35961-5e37-4e7b-bf8d-5a8d7c273893-kube-api-access-drkkz\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543276 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a14e8823-8f98-40c4-b20d-dfea320c8304-original-pull-secret\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543320 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-env-overrides\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543347 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-socket-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543376 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-k8s-cni-cncf-io\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543400 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543437 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-slash\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543453 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-bin\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.543918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.543614 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/98a7d07b-33d6-4778-8bb0-7c277468d407-secret-thanos-querier-tls\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.553308 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.553288 5591 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:44:12.553395 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:12.553338 5591 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" Nov 02 13:44:12.553774 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.553759 5591 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:44:12.553858 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:12.553800 5591 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" Nov 02 13:44:12.573706 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.573680 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8x5\" (UniqueName: \"kubernetes.io/projected/36315738-ca55-4bbb-8644-748e4ae00efe-kube-api-access-zq8x5\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.573818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.573685 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkp9k\" (UniqueName: \"kubernetes.io/projected/814f4474-5a86-4ddf-8bd8-beeebe917d71-kube-api-access-fkp9k\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.573818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.573793 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6l9\" (UniqueName: \"kubernetes.io/projected/98a7d07b-33d6-4778-8bb0-7c277468d407-kube-api-access-dg6l9\") pod \"thanos-querier-c6896987-j2hml\" (UID: \"98a7d07b-33d6-4778-8bb0-7c277468d407\") " pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:12.573930 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.573871 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzgw\" (UniqueName: \"kubernetes.io/projected/9cb24650-63d2-4d50-9a20-db9bf4a5e2c9-kube-api-access-wvzgw\") pod \"network-check-target-cc5c8\" (UID: \"9cb24650-63d2-4d50-9a20-db9bf4a5e2c9\") " pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:44:12.574324 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.574307 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4xv\" (UniqueName: \"kubernetes.io/projected/b29d373a-4cfd-4d5b-8561-e604264326db-kube-api-access-ml4xv\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:44:12.578750 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.578729 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x4f\" (UniqueName: \"kubernetes.io/projected/4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35-kube-api-access-74x4f\") pod \"prometheus-operator-9cd89b74c-jpn6p\" (UID: \"4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jpn6p" Nov 02 13:44:12.579509 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.579491 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkkz\" (UniqueName: \"kubernetes.io/projected/16e35961-5e37-4e7b-bf8d-5a8d7c273893-kube-api-access-drkkz\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.580532 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.580511 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmcr\" (UniqueName: \"kubernetes.io/projected/d5655095-e60c-40a9-9f0d-510e96cdd667-kube-api-access-tvmcr\") pod \"monitoring-plugin-77f4f76594-vdjd6\" (UID: \"d5655095-e60c-40a9-9f0d-510e96cdd667\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:12.583645 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.583619 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5g6\" (UniqueName: \"kubernetes.io/projected/5291323c-ea92-499c-b246-a4f692f3f8c5-kube-api-access-8x5g6\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:44:12.583802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.583720 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsdw\" (UniqueName: \"kubernetes.io/projected/88b783e5-08d6-44c6-ac37-b50adf8736ea-kube-api-access-8wsdw\") pod \"network-metrics-daemon-v8n7n\" (UID: \"88b783e5-08d6-44c6-ac37-b50adf8736ea\") " pod="openshift-multus/network-metrics-daemon-v8n7n" Nov 02 13:44:12.583885 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.583851 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jtm\" (UniqueName: \"kubernetes.io/projected/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-kube-api-access-79jtm\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.584179 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.584157 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrgf\" (UniqueName: \"kubernetes.io/projected/cdde073d-a59a-4953-8650-3c997810c4b9-kube-api-access-drrgf\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.584269 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.584183 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx298\" (UniqueName: \"kubernetes.io/projected/257163a1-1ead-438e-94c2-ca672bdcaa17-kube-api-access-nx298\") pod \"dns-default-6bll8\" (UID: \"257163a1-1ead-438e-94c2-ca672bdcaa17\") " pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:12.584269 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.584163 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6zn\" (UniqueName: \"kubernetes.io/projected/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-kube-api-access-4z6zn\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.584403 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.584299 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb826\" (UniqueName: \"kubernetes.io/projected/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kube-api-access-jb826\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.587906 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.587878 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xt65\" (UniqueName: \"kubernetes.io/projected/790cdff4-1780-4548-a327-e7b7df474347-kube-api-access-4xt65\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:44:12.588073 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.588058 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z52s\" (UniqueName: \"kubernetes.io/projected/0ef62365-5210-41a9-a45c-7e71cd04fb4c-kube-api-access-9z52s\") pod \"ingress-canary-j79zx\" (UID: \"0ef62365-5210-41a9-a45c-7e71cd04fb4c\") " pod="openshift-ingress-canary/ingress-canary-j79zx" Nov 02 13:44:12.644287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644261 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-netd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644292 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-etc-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644310 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-root\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644328 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-systemd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644373 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-systemd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644376 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-etc-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644396 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-root\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644374 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-netd\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644431 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-systemd-units\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644402 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-systemd-units\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644473 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644503 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644507 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-kubernetes\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644547 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-multus\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644556 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-kubernetes\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644573 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-hostroot\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644598 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-sys\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644608 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-multus\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644613 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-kubelet\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644643 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-kubelet\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644650 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644661 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-hostroot\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644672 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-sys\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644677 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-multus-certs\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644702 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644728 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-host\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644734 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-multus-certs\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644748 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644775 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644793 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-host\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644799 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-bin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644832 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysconfig\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.644835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644839 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-cni-bin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644857 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-lib-modules\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644851 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644884 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-cnibin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644911 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-log-socket\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644921 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-cnibin\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644891 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysconfig\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644936 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-conf\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644858 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644948 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-log-socket\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.644965 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-systemd\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645009 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-system-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645024 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-lib-modules\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645030 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645056 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-systemd\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645063 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645071 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-system-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645077 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-sys\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645087 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-conf\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645104 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-socket-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645120 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-sys\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645129 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-k8s-cni-cncf-io\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645164 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-slash\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645176 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-socket-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645179 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-k8s-cni-cncf-io\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645180 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-bin\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645208 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-slash\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645222 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-device-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645247 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645245 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-cni-bin\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645270 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-kubelet\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645281 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-device-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645294 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-netns\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645305 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-cni-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645312 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-var-lib-kubelet\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645317 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645341 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-var-lib-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645345 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-run-netns\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645365 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-etc-kubernetes\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645384 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b29d373a-4cfd-4d5b-8561-e604264326db-host-slash\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645398 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-sysctl-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645399 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-var-lib-openvswitch\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645407 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-netns\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645417 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-etc-kubernetes\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645431 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b29d373a-4cfd-4d5b-8561-e604264326db-host-slash\") pod \"iptables-alerter-ff8sv\" (UID: \"b29d373a-4cfd-4d5b-8561-e604264326db\") " pod="openshift-network-operator/iptables-alerter-ff8sv" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645436 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-modprobe-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645451 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-host-run-netns\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645464 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cnibin\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645484 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-conf-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645487 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-etc-modprobe-d\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645504 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-run\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645510 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-conf-dir\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645526 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-sys-fs\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645531 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-cnibin\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645549 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-run\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645554 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-node-log\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645574 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-node-log\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645576 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-sys-fs\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645580 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/790cdff4-1780-4548-a327-e7b7df474347-hosts-file\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645605 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-registration-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645624 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-kubelet-config\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645636 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/790cdff4-1780-4548-a327-e7b7df474347-hosts-file\") pod \"node-resolver-j9lm6\" (UID: \"790cdff4-1780-4548-a327-e7b7df474347\") " pod="openshift-dns/node-resolver-j9lm6" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645665 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33fad09e-a1b4-4998-bb14-8e6e33a3a03c-registration-dir\") pod \"aws-ebs-csi-driver-node-bwps4\" (UID: \"33fad09e-a1b4-4998-bb14-8e6e33a3a03c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwps4" Nov 02 13:44:12.645642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645637 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-dbus\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645688 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-wtmp\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645706 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-var-lib-kubelet\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645720 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5291323c-ea92-499c-b246-a4f692f3f8c5-host\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645741 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645670 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-kubelet-config\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645768 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645773 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5291323c-ea92-499c-b246-a4f692f3f8c5-host\") pod \"node-ca-9d7n9\" (UID: \"5291323c-ea92-499c-b246-a4f692f3f8c5\") " pod="openshift-image-registry/node-ca-9d7n9" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645672 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a14e8823-8f98-40c4-b20d-dfea320c8304-dbus\") pod \"global-pull-secret-syncer-scx6x\" (UID: \"a14e8823-8f98-40c4-b20d-dfea320c8304\") " pod="kube-system/global-pull-secret-syncer-scx6x" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645792 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d8e273bc-9f51-4993-86c2-7e02c3b88c7d-node-exporter-wtmp\") pod \"node-exporter-sjj7t\" (UID: \"d8e273bc-9f51-4993-86c2-7e02c3b88c7d\") " pod="openshift-monitoring/node-exporter-sjj7t" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645800 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36315738-ca55-4bbb-8644-748e4ae00efe-var-lib-kubelet\") pod \"tuned-ttzxz\" (UID: \"36315738-ca55-4bbb-8644-748e4ae00efe\") " pod="openshift-cluster-node-tuning-operator/tuned-ttzxz" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645809 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-socket-dir-parent\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645833 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/814f4474-5a86-4ddf-8bd8-beeebe917d71-multus-socket-dir-parent\") pod \"multus-g268n\" (UID: \"814f4474-5a86-4ddf-8bd8-beeebe917d71\") " pod="openshift-multus/multus-g268n" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645846 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cdde073d-a59a-4953-8650-3c997810c4b9-crio-socket\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645871 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-ovn\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645892 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cdde073d-a59a-4953-8650-3c997810c4b9-crio-socket\") pod \"insights-runtime-extractor-87kgf\" (UID: \"cdde073d-a59a-4953-8650-3c997810c4b9\") " pod="openshift-insights/insights-runtime-extractor-87kgf" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645896 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-system-cni-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645931 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52eda2b9-8258-4f7e-bc72-6ad9f6f1167a-run-ovn\") pod \"ovnkube-node-mq8vn\" (UID: \"52eda2b9-8258-4f7e-bc72-6ad9f6f1167a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:12.647465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:12.645956 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16e35961-5e37-4e7b-bf8d-5a8d7c273893-system-cni-dir\") pod \"multus-additional-cni-plugins-vbv78\" (UID: \"16e35961-5e37-4e7b-bf8d-5a8d7c273893\") " pod="openshift-multus/multus-additional-cni-plugins-vbv78" Nov 02 13:44:13.367272 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:13.367231 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:13.370031 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:13.370005 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:13.370802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:13.370784 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vdjd6" Nov 02 13:44:13.452409 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:13.452383 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8hhsr" Nov 02 13:44:13.539355 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:13.539316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:13.741606 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:13.741573 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:13.909354 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:13.909318 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:14.091136 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:14.091047 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:14.733413 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:14.733382 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:14.737610 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:14.737589 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c6896987-j2hml" Nov 02 13:44:14.848796 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:14.848758 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6bll8" Nov 02 13:44:15.969772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:15.969738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:16.134366 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:16.134332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:16.524448 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:16.524395 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:44:17.590216 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:17.590190 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:17.773974 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:17.773942 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:17.952593 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:17.952564 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:19.225585 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:19.225543 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:19.418108 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:19.418053 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:21.782317 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:21.782278 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:21.782770 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:21.782560 5591 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:44:22.585339 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:22.585285 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:22.747378 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:22.747343 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:28.705372 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:28.705319 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:28.864849 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:28.864817 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:31.071564 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:44:31.071530 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:44:38.988817 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:38.988787 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mq8vn" Nov 02 13:44:43.725315 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:43.725277 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:43.911961 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:43.911927 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:46.527023 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:46.526969 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cc5c8" Nov 02 13:44:47.617649 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:47.617613 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:47.787502 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:47.787472 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:52.836884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.836847 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69ff98d85d-s4mqk"] Nov 02 13:44:52.839062 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.839041 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.868803 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.868770 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 13:44:52.868803 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.868775 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 13:44:52.869062 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.868808 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 13:44:52.869062 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.868833 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 13:44:52.869062 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.868856 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2nt62\"" Nov 02 13:44:52.869428 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.869408 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69ff98d85d-s4mqk"] Nov 02 13:44:52.908997 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.908965 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.909137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.909013 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-metrics-client-ca\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.909137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.909032 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znn6\" (UniqueName: \"kubernetes.io/projected/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-kube-api-access-6znn6\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.909264 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.909165 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.909264 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.909202 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-telemeter-client-tls\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.909264 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.909221 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-secret-telemeter-client\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.909352 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.909285 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-federate-client-tls\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.909352 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.909340 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-serving-certs-ca-bundle\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:52.924061 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.924036 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 13:44:52.937193 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:52.937174 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 13:44:53.010429 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010393 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.010429 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010430 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-telemeter-client-tls\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.010627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010448 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-secret-telemeter-client\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.010627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010474 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-federate-client-tls\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.010627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010519 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-serving-certs-ca-bundle\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.010627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010552 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.010627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010583 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-metrics-client-ca\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.010627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010610 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znn6\" (UniqueName: \"kubernetes.io/projected/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-kube-api-access-6znn6\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.010918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.010887 5591 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 13:44:53.011390 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.011356 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-serving-certs-ca-bundle\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.011725 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.011695 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.011833 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.011780 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-metrics-client-ca\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.013631 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.013606 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-federate-client-tls\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.013631 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.013627 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-telemeter-client-tls\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.013829 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.013809 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-secret-telemeter-client\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.013888 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.013806 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.034583 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.034555 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znn6\" (UniqueName: \"kubernetes.io/projected/3f56563e-0aeb-46ab-9f7f-ae0f6409ec33-kube-api-access-6znn6\") pod \"telemeter-client-69ff98d85d-s4mqk\" (UID: \"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33\") " pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.154442 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.154406 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" Nov 02 13:44:53.483424 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.483346 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69ff98d85d-s4mqk"] Nov 02 13:44:53.489719 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:44:53.489677 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f56563e_0aeb_46ab_9f7f_ae0f6409ec33.slice/crio-a70914ea8e85ff9d69db11b9ebcbcab5f8698d2d5d676f3eaf80bf4e25df3db8 WatchSource:0}: Error finding container a70914ea8e85ff9d69db11b9ebcbcab5f8698d2d5d676f3eaf80bf4e25df3db8: Status 404 returned error can't find the container with id a70914ea8e85ff9d69db11b9ebcbcab5f8698d2d5d676f3eaf80bf4e25df3db8 Nov 02 13:44:53.493629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.492407 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 13:44:53.659480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:53.659433 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" event={"ID":"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33","Type":"ContainerStarted","Data":"a70914ea8e85ff9d69db11b9ebcbcab5f8698d2d5d676f3eaf80bf4e25df3db8"} Nov 02 13:44:55.666718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:55.666682 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" event={"ID":"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33","Type":"ContainerStarted","Data":"678ba88695f6397cc138b351db30a4d4f0dd8fd29fa36462201190773064355a"} Nov 02 13:44:56.671501 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:56.671459 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" event={"ID":"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33","Type":"ContainerStarted","Data":"0850682352c7bacedbc9d2834e75d49a1b56aa852c0c717b08b2ae714a0d8686"} Nov 02 13:44:56.671906 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:56.671524 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" event={"ID":"3f56563e-0aeb-46ab-9f7f-ae0f6409ec33","Type":"ContainerStarted","Data":"65e51dec3cae676a43649b2a551064d8b77f5b717844561cd24336f5797e2446"} Nov 02 13:44:56.746460 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:56.746411 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69ff98d85d-s4mqk" podStartSLOduration=1.9983391510000001 podStartE2EDuration="4.746397953s" totalImagesPullingTime="2.748058802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:56.734398409 +0000 UTC m=+45.460020178" watchObservedRunningTime="2026-11-02 13:44:56.746397953 +0000 UTC m=+45.472019716" Nov 02 13:44:57.090228 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:57.090138 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:57.251843 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:57.251811 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:57.454764 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:44:57.454724 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:01.071953 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:45:01.071914 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:45:19.381818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:19.381778 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:31.072370 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:45:31.072331 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:45:43.464886 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:43.464852 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:43.629016 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:43.628968 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:51.083447 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:51.083410 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:51.250826 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:51.250792 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:51.424151 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:51.424113 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:56.129519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:56.129439 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:56.336050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:56.336017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:57.929951 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:57.929919 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:58.715314 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:58.715280 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:58.884893 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:45:58.884862 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:01.072470 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:46:01.072432 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:46:17.812476 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:46:17.812436 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:19.037165 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:46:19.037128 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:19.457556 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:46:19.457524 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:31.072321 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:46:31.072287 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:47:01.072487 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:47:01.072448 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:47:19.570052 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:19.570018 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:31.072590 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:47:31.072508 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:47:52.483624 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:52.483581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:54.122851 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.122809 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:54.431738 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.431698 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5"] Nov 02 13:47:54.434729 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.434713 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.452818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.452793 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 13:47:54.453021 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.453004 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 13:47:54.453066 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.453032 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 13:47:54.464890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.464872 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 13:47:54.464953 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.464909 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 13:47:54.464953 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.464944 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 13:47:54.465030 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.464949 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5wglw\"" Nov 02 13:47:54.476651 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.476630 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5"] Nov 02 13:47:54.520511 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.520480 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:54.575625 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.575591 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.575780 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.575642 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.575780 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.575715 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5bgx\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-kube-api-access-r5bgx\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.575780 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.575763 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a77acb-04be-4ff6-97ee-b31b18a0700e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.575895 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.575837 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.575895 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.575862 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.575895 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.575878 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.676441 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.676405 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.676441 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.676446 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.676660 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.676466 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.676660 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.676490 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.676660 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.676631 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.676809 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.676668 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5bgx\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-kube-api-access-r5bgx\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.676809 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.676698 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a77acb-04be-4ff6-97ee-b31b18a0700e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.677177 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.677142 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.678877 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.678853 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a77acb-04be-4ff6-97ee-b31b18a0700e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.678877 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.678865 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.679078 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.678970 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.679078 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.679036 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.701473 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.701412 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.701578 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.701537 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5bgx\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-kube-api-access-r5bgx\") pod \"istiod-openshift-gateway-7876bfcb74-65kv5\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:54.724924 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.724896 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:54.745907 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:54.745886 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:55.075118 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:55.075032 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5"] Nov 02 13:47:55.078338 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:47:55.078311 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a77acb_04be_4ff6_97ee_b31b18a0700e.slice/crio-b8a37cf72498b0e15d72b23a02e789455b6b25e48ed56cb214c62f0fc5333369 WatchSource:0}: Error finding container b8a37cf72498b0e15d72b23a02e789455b6b25e48ed56cb214c62f0fc5333369: Status 404 returned error can't find the container with id b8a37cf72498b0e15d72b23a02e789455b6b25e48ed56cb214c62f0fc5333369 Nov 02 13:47:55.229321 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:55.229279 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" event={"ID":"21a77acb-04be-4ff6-97ee-b31b18a0700e","Type":"ContainerStarted","Data":"b8a37cf72498b0e15d72b23a02e789455b6b25e48ed56cb214c62f0fc5333369"} Nov 02 13:47:56.149254 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:56.149219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:57.479569 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:57.479536 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:47:57.479851 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:57.479599 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:47:58.211515 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:58.211458 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:58.253291 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:58.253253 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" event={"ID":"21a77acb-04be-4ff6-97ee-b31b18a0700e","Type":"ContainerStarted","Data":"f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047"} Nov 02 13:47:58.253491 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:58.253471 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:58.254920 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:58.254892 5591 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-65kv5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 13:47:58.255038 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:58.254952 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" podUID="21a77acb-04be-4ff6-97ee-b31b18a0700e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 13:47:58.328326 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:58.328275 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" podStartSLOduration=1.9297295989999999 podStartE2EDuration="4.328263273s" totalImagesPullingTime="2.398533674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:47:58.315871076 +0000 UTC m=+227.041492846" watchObservedRunningTime="2026-11-02 13:47:58.328263273 +0000 UTC m=+227.053885042" Nov 02 13:47:58.426749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:58.426714 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:59.257503 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:59.257473 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:47:59.492738 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:59.492699 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:59.669906 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:47:59.669872 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:01.072352 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:48:01.072314 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:48:02.793127 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:02.793088 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:07.895230 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:07.895199 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:08.055526 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:08.055492 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:19.694461 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:19.694427 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:31.072100 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:48:31.072059 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:48:31.304903 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:31.304871 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:32.527533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:32.527502 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:33.578619 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:33.578574 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:36.731401 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:36.731366 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:36.896041 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:36.896014 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:39.264606 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:39.264575 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:39.510601 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:39.510571 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:39.689195 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:39.689160 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:40.365483 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.365446 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:40.546596 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.546559 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b"] Nov 02 13:48:40.549775 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.549757 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.607309 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.607278 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.607456 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.607321 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bpj\" (UniqueName: \"kubernetes.io/projected/98e74af6-b2be-4414-9801-0459b4791cc7-kube-api-access-k4bpj\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.607456 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.607341 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.607533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.607458 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/98e74af6-b2be-4414-9801-0459b4791cc7-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.607533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.607507 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e74af6-b2be-4414-9801-0459b4791cc7-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.607533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.607525 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.649931 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.649898 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 13:48:40.649931 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.649906 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 13:48:40.665391 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.665350 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b"] Nov 02 13:48:40.667221 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.667186 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-drxxd\"" Nov 02 13:48:40.668761 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.668744 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 13:48:40.668977 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.668958 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 13:48:40.669035 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.669011 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 13:48:40.708509 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.708474 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.708631 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.708524 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bpj\" (UniqueName: \"kubernetes.io/projected/98e74af6-b2be-4414-9801-0459b4791cc7-kube-api-access-k4bpj\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.708670 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.708634 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.708718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.708691 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/98e74af6-b2be-4414-9801-0459b4791cc7-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.708772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.708731 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e74af6-b2be-4414-9801-0459b4791cc7-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.708772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.708758 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.709115 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.709087 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/98e74af6-b2be-4414-9801-0459b4791cc7-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.711076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.711051 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.711194 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.711060 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98e74af6-b2be-4414-9801-0459b4791cc7-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.711281 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.711261 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.711339 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.711263 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98e74af6-b2be-4414-9801-0459b4791cc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.740154 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.740120 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bpj\" (UniqueName: \"kubernetes.io/projected/98e74af6-b2be-4414-9801-0459b4791cc7-kube-api-access-k4bpj\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-jvf4b\" (UID: \"98e74af6-b2be-4414-9801-0459b4791cc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:40.861779 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.861742 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:40.863727 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:40.863709 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:41.063398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:41.063364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:41.237786 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:41.237712 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b"] Nov 02 13:48:41.241417 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:48:41.241388 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e74af6_b2be_4414_9801_0459b4791cc7.slice/crio-60e360556dc82b411b29434425b435864ddc94d08a0b3c386946e6dc52263967 WatchSource:0}: Error finding container 60e360556dc82b411b29434425b435864ddc94d08a0b3c386946e6dc52263967: Status 404 returned error can't find the container with id 60e360556dc82b411b29434425b435864ddc94d08a0b3c386946e6dc52263967 Nov 02 13:48:41.297098 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:41.297058 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:41.391311 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:41.391276 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" event={"ID":"98e74af6-b2be-4414-9801-0459b4791cc7","Type":"ContainerStarted","Data":"60e360556dc82b411b29434425b435864ddc94d08a0b3c386946e6dc52263967"} Nov 02 13:48:41.556615 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:41.556536 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:41.874418 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:41.874320 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:43.788618 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:43.788582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:44.402978 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:44.402943 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" event={"ID":"98e74af6-b2be-4414-9801-0459b4791cc7","Type":"ContainerStarted","Data":"f001ca0af37878eba7ba266852d856eea56f46c44d603f2cda4232b72840f4b3"} Nov 02 13:48:45.408199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:45.408165 5591 generic.go:356] "Generic (PLEG): container finished" podID="98e74af6-b2be-4414-9801-0459b4791cc7" containerID="f001ca0af37878eba7ba266852d856eea56f46c44d603f2cda4232b72840f4b3" exitCode=0 Nov 02 13:48:45.408606 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:45.408208 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" event={"ID":"98e74af6-b2be-4414-9801-0459b4791cc7","Type":"ContainerDied","Data":"f001ca0af37878eba7ba266852d856eea56f46c44d603f2cda4232b72840f4b3"} Nov 02 13:48:46.413423 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:46.413387 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" event={"ID":"98e74af6-b2be-4414-9801-0459b4791cc7","Type":"ContainerStarted","Data":"ca901069096a45a7274f6e63110a9fb1aa2945786fef00e6029b764b32c7b9b7"} Nov 02 13:48:46.413821 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:46.413533 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:46.570535 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:46.570482 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" podStartSLOduration=3.015689734 podStartE2EDuration="6.570468759s" totalImagesPullingTime="3.140866187s" totalInitContainerRuntime="413.912838ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:46.557935938 +0000 UTC m=+275.283557705" watchObservedRunningTime="2026-11-02 13:48:46.570468759 +0000 UTC m=+275.296090529" Nov 02 13:48:46.640513 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:46.640477 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:46.830279 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:46.830198 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:49.858365 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:49.858331 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:51.980130 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:51.980092 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:52.245412 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:52.245326 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:57.420192 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:57.420114 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-jvf4b" Nov 02 13:48:58.020312 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:58.020279 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:58.836437 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:48:58.836401 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:01.072257 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:49:01.072220 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:49:02.093535 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:02.093496 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:02.407285 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:02.407249 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:02.686186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:02.686114 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:02.879399 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:02.879367 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:03.202829 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:03.202803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:04.125285 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:04.125249 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:06.659204 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:06.659171 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:08.197577 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.197540 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286"] Nov 02 13:49:08.200749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.200731 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" Nov 02 13:49:08.216332 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.216310 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-d9l6h\"" Nov 02 13:49:08.228515 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.228491 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 13:49:08.228949 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.228926 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 13:49:08.229075 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.228955 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 13:49:08.230012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.229973 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286"] Nov 02 13:49:08.338209 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.338167 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdb8\" (UniqueName: \"kubernetes.io/projected/d529065b-33fe-4288-b543-a51b5de16f88-kube-api-access-xhdb8\") pod \"dns-operator-controller-manager-8557cbdb84-sj286\" (UID: \"d529065b-33fe-4288-b543-a51b5de16f88\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" Nov 02 13:49:08.439127 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.439090 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdb8\" (UniqueName: \"kubernetes.io/projected/d529065b-33fe-4288-b543-a51b5de16f88-kube-api-access-xhdb8\") pod \"dns-operator-controller-manager-8557cbdb84-sj286\" (UID: \"d529065b-33fe-4288-b543-a51b5de16f88\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" Nov 02 13:49:08.472582 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.472514 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdb8\" (UniqueName: \"kubernetes.io/projected/d529065b-33fe-4288-b543-a51b5de16f88-kube-api-access-xhdb8\") pod \"dns-operator-controller-manager-8557cbdb84-sj286\" (UID: \"d529065b-33fe-4288-b543-a51b5de16f88\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" Nov 02 13:49:08.513119 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.513090 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" Nov 02 13:49:08.592190 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.592152 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:08.864680 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.864603 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286"] Nov 02 13:49:08.867452 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:08.867413 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:08.867840 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:49:08.867818 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd529065b_33fe_4288_b543_a51b5de16f88.slice/crio-c07ac720114535d60d8311087ee1e4cb7e97e649e53999d495f79781a6f2878c WatchSource:0}: Error finding container c07ac720114535d60d8311087ee1e4cb7e97e649e53999d495f79781a6f2878c: Status 404 returned error can't find the container with id c07ac720114535d60d8311087ee1e4cb7e97e649e53999d495f79781a6f2878c Nov 02 13:49:09.411437 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:09.411399 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:09.488886 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:09.488844 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" event={"ID":"d529065b-33fe-4288-b543-a51b5de16f88","Type":"ContainerStarted","Data":"c07ac720114535d60d8311087ee1e4cb7e97e649e53999d495f79781a6f2878c"} Nov 02 13:49:09.707010 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:09.706909 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:10.235112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.235075 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:10.585848 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.585767 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh"] Nov 02 13:49:10.596085 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.596048 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" Nov 02 13:49:10.633495 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.633462 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nl525\"" Nov 02 13:49:10.646974 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.646936 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh"] Nov 02 13:49:10.758641 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.758604 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49grt\" (UniqueName: \"kubernetes.io/projected/c7a64f22-0110-479d-900e-b27ee83b6985-kube-api-access-49grt\") pod \"limitador-operator-controller-manager-5778bb4449-sxzjh\" (UID: \"c7a64f22-0110-479d-900e-b27ee83b6985\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" Nov 02 13:49:10.859160 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.859077 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49grt\" (UniqueName: \"kubernetes.io/projected/c7a64f22-0110-479d-900e-b27ee83b6985-kube-api-access-49grt\") pod \"limitador-operator-controller-manager-5778bb4449-sxzjh\" (UID: \"c7a64f22-0110-479d-900e-b27ee83b6985\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" Nov 02 13:49:10.891963 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.891926 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49grt\" (UniqueName: \"kubernetes.io/projected/c7a64f22-0110-479d-900e-b27ee83b6985-kube-api-access-49grt\") pod \"limitador-operator-controller-manager-5778bb4449-sxzjh\" (UID: \"c7a64f22-0110-479d-900e-b27ee83b6985\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" Nov 02 13:49:10.912210 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:10.912176 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" Nov 02 13:49:11.225021 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:11.224968 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh"] Nov 02 13:49:11.229655 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:49:11.229626 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a64f22_0110_479d_900e_b27ee83b6985.slice/crio-6ecfbb27812502f04689045918a9f413f030f5f681792eb9450866328a29335e WatchSource:0}: Error finding container 6ecfbb27812502f04689045918a9f413f030f5f681792eb9450866328a29335e: Status 404 returned error can't find the container with id 6ecfbb27812502f04689045918a9f413f030f5f681792eb9450866328a29335e Nov 02 13:49:11.473113 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:11.473041 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:49:11.473289 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:11.473275 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:49:11.498533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:11.498500 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" event={"ID":"d529065b-33fe-4288-b543-a51b5de16f88","Type":"ContainerStarted","Data":"a3636d483b76cc4808782826d294e4d92dbee51351c87e0792be3360f9eb4abe"} Nov 02 13:49:11.498693 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:11.498674 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" Nov 02 13:49:11.499701 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:11.499674 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" event={"ID":"c7a64f22-0110-479d-900e-b27ee83b6985","Type":"ContainerStarted","Data":"6ecfbb27812502f04689045918a9f413f030f5f681792eb9450866328a29335e"} Nov 02 13:49:11.556039 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:11.555948 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" podStartSLOduration=1.273214715 podStartE2EDuration="3.555932836s" totalImagesPullingTime="2.282718121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:11.550714012 +0000 UTC m=+300.276335780" watchObservedRunningTime="2026-11-02 13:49:11.555932836 +0000 UTC m=+300.281554604" Nov 02 13:49:11.802512 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:11.802437 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:12.010790 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:12.010748 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:12.357145 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:12.357111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:12.691999 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:12.691953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:13.508915 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:13.508882 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" event={"ID":"c7a64f22-0110-479d-900e-b27ee83b6985","Type":"ContainerStarted","Data":"fc66d738e7d1e9413b532d14b8ade670e23a229429f83ada32f403e1811e2ce4"} Nov 02 13:49:13.509317 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:13.508974 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" Nov 02 13:49:13.552976 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:13.552926 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" podStartSLOduration=2.250349646 podStartE2EDuration="3.552913957s" totalImagesPullingTime="1.302564311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:13.552673997 +0000 UTC m=+302.278295767" watchObservedRunningTime="2026-11-02 13:49:13.552913957 +0000 UTC m=+302.278535727" Nov 02 13:49:16.528662 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:16.528630 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:16.786526 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:16.786449 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:17.166821 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:17.166793 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:19.814513 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:19.814473 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:22.506448 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:22.506416 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sj286" Nov 02 13:49:23.170488 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:23.170451 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:23.410370 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:23.410340 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:24.514526 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:24.514493 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sxzjh" Nov 02 13:49:25.916311 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:25.916274 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:26.187800 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.187714 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:26.384726 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.384691 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:26.770184 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.770144 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc"] Nov 02 13:49:26.772207 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.772191 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:26.802718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.802690 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc"] Nov 02 13:49:26.815831 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.815804 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:26.857533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.857505 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 13:49:26.857699 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.857616 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4tb9q\"" Nov 02 13:49:26.857699 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.857683 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 13:49:26.874871 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.874830 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0c62419-de89-4af5-951b-52a33f4ac5b1-tmp\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:26.875017 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.874939 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswdv\" (UniqueName: \"kubernetes.io/projected/a0c62419-de89-4af5-951b-52a33f4ac5b1-kube-api-access-jswdv\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:26.875017 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.875003 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c62419-de89-4af5-951b-52a33f4ac5b1-tls-certs\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:26.976256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.976224 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0c62419-de89-4af5-951b-52a33f4ac5b1-tmp\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:26.976612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.976286 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswdv\" (UniqueName: \"kubernetes.io/projected/a0c62419-de89-4af5-951b-52a33f4ac5b1-kube-api-access-jswdv\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:26.976612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.976322 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c62419-de89-4af5-951b-52a33f4ac5b1-tls-certs\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:26.978462 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.978434 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0c62419-de89-4af5-951b-52a33f4ac5b1-tmp\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:26.979055 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:26.979038 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c62419-de89-4af5-951b-52a33f4ac5b1-tls-certs\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:27.002366 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.002344 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswdv\" (UniqueName: \"kubernetes.io/projected/a0c62419-de89-4af5-951b-52a33f4ac5b1-kube-api-access-jswdv\") pod \"kube-auth-proxy-6fd64f76dd-9f2pc\" (UID: \"a0c62419-de89-4af5-951b-52a33f4ac5b1\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:27.036499 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.036403 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:27.084844 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.084809 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" Nov 02 13:49:27.243895 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.243859 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:27.386242 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.386210 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc"] Nov 02 13:49:27.389829 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:49:27.389802 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c62419_de89_4af5_951b_52a33f4ac5b1.slice/crio-78349522aa1a190b7267fdfbfe2fc2a6d49d7cd37a8e91cc5b2f0e944780bac3 WatchSource:0}: Error finding container 78349522aa1a190b7267fdfbfe2fc2a6d49d7cd37a8e91cc5b2f0e944780bac3: Status 404 returned error can't find the container with id 78349522aa1a190b7267fdfbfe2fc2a6d49d7cd37a8e91cc5b2f0e944780bac3 Nov 02 13:49:27.435942 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.435908 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:27.558688 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.558647 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" event={"ID":"a0c62419-de89-4af5-951b-52a33f4ac5b1","Type":"ContainerStarted","Data":"78349522aa1a190b7267fdfbfe2fc2a6d49d7cd37a8e91cc5b2f0e944780bac3"} Nov 02 13:49:27.619664 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.619633 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:27.861508 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:27.861434 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:28.109213 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:28.109172 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:28.456377 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:28.456345 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:28.997409 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:28.997373 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:29.170657 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.170621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:29.498055 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.498021 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:49:29.500967 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.500944 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:29.520620 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.520593 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 13:49:29.533076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.533045 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:49:29.535699 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.535676 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t7pr2\"" Nov 02 13:49:29.569520 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.569487 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:49:29.598048 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.598009 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c4e7da5-37b7-45bd-9840-6c87ab416429-config-file\") pod \"limitador-limitador-7c9b87d574-4zfwv\" (UID: \"1c4e7da5-37b7-45bd-9840-6c87ab416429\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:29.598224 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.598134 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7njpg\" (UniqueName: \"kubernetes.io/projected/1c4e7da5-37b7-45bd-9840-6c87ab416429-kube-api-access-7njpg\") pod \"limitador-limitador-7c9b87d574-4zfwv\" (UID: \"1c4e7da5-37b7-45bd-9840-6c87ab416429\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:29.698729 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.698510 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c4e7da5-37b7-45bd-9840-6c87ab416429-config-file\") pod \"limitador-limitador-7c9b87d574-4zfwv\" (UID: \"1c4e7da5-37b7-45bd-9840-6c87ab416429\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:29.698729 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.698580 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7njpg\" (UniqueName: \"kubernetes.io/projected/1c4e7da5-37b7-45bd-9840-6c87ab416429-kube-api-access-7njpg\") pod \"limitador-limitador-7c9b87d574-4zfwv\" (UID: \"1c4e7da5-37b7-45bd-9840-6c87ab416429\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:29.699225 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.699206 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c4e7da5-37b7-45bd-9840-6c87ab416429-config-file\") pod \"limitador-limitador-7c9b87d574-4zfwv\" (UID: \"1c4e7da5-37b7-45bd-9840-6c87ab416429\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:29.721404 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.721382 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7njpg\" (UniqueName: \"kubernetes.io/projected/1c4e7da5-37b7-45bd-9840-6c87ab416429-kube-api-access-7njpg\") pod \"limitador-limitador-7c9b87d574-4zfwv\" (UID: \"1c4e7da5-37b7-45bd-9840-6c87ab416429\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:29.813813 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:29.813732 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:30.069851 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:30.069776 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:30.142725 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:30.142691 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:49:30.147712 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:49:30.147688 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4e7da5_37b7_45bd_9840_6c87ab416429.slice/crio-ed663187d24167970dafc48204b392d82b7cc3174a5d02217880b750b84de5fd WatchSource:0}: Error finding container ed663187d24167970dafc48204b392d82b7cc3174a5d02217880b750b84de5fd: Status 404 returned error can't find the container with id ed663187d24167970dafc48204b392d82b7cc3174a5d02217880b750b84de5fd Nov 02 13:49:30.573307 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:30.573257 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" event={"ID":"1c4e7da5-37b7-45bd-9840-6c87ab416429","Type":"ContainerStarted","Data":"ed663187d24167970dafc48204b392d82b7cc3174a5d02217880b750b84de5fd"} Nov 02 13:49:30.575281 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:30.575247 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" event={"ID":"a0c62419-de89-4af5-951b-52a33f4ac5b1","Type":"ContainerStarted","Data":"0c1921148cf4a019f98ae98c3d78e3942e8de9ddb8884d9a102b3c5f18425eac"} Nov 02 13:49:30.629908 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:30.629847 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-9f2pc" podStartSLOduration=2.358336594 podStartE2EDuration="4.629828018s" totalImagesPullingTime="2.271491424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:30.629398952 +0000 UTC m=+319.355020745" watchObservedRunningTime="2026-11-02 13:49:30.629828018 +0000 UTC m=+319.355449789" Nov 02 13:49:30.764878 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:30.764839 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:30.952074 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:30.952049 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:31.072132 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:49:31.072091 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:49:31.581200 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:31.581161 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" event={"ID":"1c4e7da5-37b7-45bd-9840-6c87ab416429","Type":"ContainerStarted","Data":"f0a2b061e41065f1fd1b7eea0e4512d5db71a4399302df6274ca5bf298f4a7f6"} Nov 02 13:49:31.581622 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:31.581362 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:31.622861 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:31.622818 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" podStartSLOduration=1.883409474 podStartE2EDuration="2.622805671s" totalImagesPullingTime="739.396197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:31.62236119 +0000 UTC m=+320.347982960" watchObservedRunningTime="2026-11-02 13:49:31.622805671 +0000 UTC m=+320.348427435" Nov 02 13:49:36.302404 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:36.302367 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:36.468663 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:36.468624 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:36.700027 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:36.699994 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:36.867388 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:36.867359 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:37.088944 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:37.088801 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:37.247471 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:37.247426 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:42.578274 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:42.578242 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:42.586825 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:42.586801 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4zfwv" Nov 02 13:49:42.743919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:42.743885 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:43.214108 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:43.214073 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:43.377347 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:43.377315 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:47.314899 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:47.314868 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:47.497539 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:47.497478 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:48.584041 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:48.584004 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:48.827159 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:48.827124 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:49.379772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:49:49.379743 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:00.840510 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:00.840470 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:01.072219 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:50:01.072184 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:50:19.385765 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.385721 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:19.410678 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.410643 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-pl7vn"] Nov 02 13:50:19.417877 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.417848 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.435466 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.435431 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jtrps\"" Nov 02 13:50:19.436214 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.436191 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-pl7vn"] Nov 02 13:50:19.447521 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.447495 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 13:50:19.447521 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.447513 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 13:50:19.447715 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.447598 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 13:50:19.447715 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.447680 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 13:50:19.447825 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.447765 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 13:50:19.500676 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.500644 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-metrics-certs\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.500826 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.500704 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ww8k\" (UniqueName: \"kubernetes.io/projected/8b854bc5-df65-4c58-a7bc-30540708f129-kube-api-access-8ww8k\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.500826 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.500806 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-webhook-cert\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.555959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.555921 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:19.601838 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.601800 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-webhook-cert\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.602059 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.601844 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-metrics-certs\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.602059 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.601891 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww8k\" (UniqueName: \"kubernetes.io/projected/8b854bc5-df65-4c58-a7bc-30540708f129-kube-api-access-8ww8k\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.604307 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.604281 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-webhook-cert\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.604414 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.604358 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-metrics-certs\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.628507 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.628477 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww8k\" (UniqueName: \"kubernetes.io/projected/8b854bc5-df65-4c58-a7bc-30540708f129-kube-api-access-8ww8k\") pod \"maas-controller-68c8d8c8bb-pl7vn\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.731226 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.731141 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:19.981355 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:19.981253 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:20.029631 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:20.029595 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-pl7vn"] Nov 02 13:50:20.032673 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:50:20.032650 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b854bc5_df65_4c58_a7bc_30540708f129.slice/crio-38865d5fb978f1a148423a6e2f39524f4487fd90eabaa425fd8654e536b923a6 WatchSource:0}: Error finding container 38865d5fb978f1a148423a6e2f39524f4487fd90eabaa425fd8654e536b923a6: Status 404 returned error can't find the container with id 38865d5fb978f1a148423a6e2f39524f4487fd90eabaa425fd8654e536b923a6 Nov 02 13:50:20.033959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:20.033940 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 13:50:20.158078 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:20.158043 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:20.765118 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:20.765038 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" event={"ID":"8b854bc5-df65-4c58-a7bc-30540708f129","Type":"ContainerStarted","Data":"38865d5fb978f1a148423a6e2f39524f4487fd90eabaa425fd8654e536b923a6"} Nov 02 13:50:22.635117 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:22.635078 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:22.780572 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:22.780488 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" event={"ID":"8b854bc5-df65-4c58-a7bc-30540708f129","Type":"ContainerStarted","Data":"3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748"} Nov 02 13:50:22.780750 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:22.780624 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:22.852552 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:22.852493 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" podStartSLOduration=1.56917861 podStartE2EDuration="3.852479613s" totalImagesPullingTime="2.283301003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:22.851470771 +0000 UTC m=+371.577092542" watchObservedRunningTime="2026-11-02 13:50:22.852479613 +0000 UTC m=+371.578101384" Nov 02 13:50:22.942559 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:22.942478 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:25.171167 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:25.171131 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:25.382074 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:25.382039 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:28.413420 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.413387 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:28.585832 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.585795 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:28.681213 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.681130 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk"] Nov 02 13:50:28.684186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.684169 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.709925 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.709898 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk"] Nov 02 13:50:28.782610 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.782574 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.782610 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.782617 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3247735-5d35-43ac-a9cd-48c53d43ab19-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.782859 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.782641 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.782859 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.782778 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.782859 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.782834 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.782859 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.782851 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5hk\" (UniqueName: \"kubernetes.io/projected/f3247735-5d35-43ac-a9cd-48c53d43ab19-kube-api-access-hh5hk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.783051 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.782869 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.783895 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.783871 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:28.883577 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.883543 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.883762 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.883587 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3247735-5d35-43ac-a9cd-48c53d43ab19-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.883762 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.883611 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.883762 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.883742 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.884003 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.883834 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.884003 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.883871 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5hk\" (UniqueName: \"kubernetes.io/projected/f3247735-5d35-43ac-a9cd-48c53d43ab19-kube-api-access-hh5hk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.884003 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.883903 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.884643 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.884609 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.886321 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.886299 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3247735-5d35-43ac-a9cd-48c53d43ab19-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.886427 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.886350 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.886427 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.886366 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.886657 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.886638 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.907011 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.906963 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5hk\" (UniqueName: \"kubernetes.io/projected/f3247735-5d35-43ac-a9cd-48c53d43ab19-kube-api-access-hh5hk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.907585 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.907561 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3247735-5d35-43ac-a9cd-48c53d43ab19-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tnmvk\" (UID: \"f3247735-5d35-43ac-a9cd-48c53d43ab19\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:28.953299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.953217 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:28.996329 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:28.996300 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:29.285237 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:29.285162 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:29.309760 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:29.309729 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk"] Nov 02 13:50:29.313151 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:50:29.313129 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3247735_5d35_43ac_a9cd_48c53d43ab19.slice/crio-ce54f010e566d1a82c68a56ee6734bd12f01ed50236d2dca58e8fde734b89bdf WatchSource:0}: Error finding container ce54f010e566d1a82c68a56ee6734bd12f01ed50236d2dca58e8fde734b89bdf: Status 404 returned error can't find the container with id ce54f010e566d1a82c68a56ee6734bd12f01ed50236d2dca58e8fde734b89bdf Nov 02 13:50:29.315126 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:29.315100 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:50:29.315198 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:29.315162 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:50:29.814102 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:29.814045 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" event={"ID":"f3247735-5d35-43ac-a9cd-48c53d43ab19","Type":"ContainerStarted","Data":"94560d8f3ef97310517d7f9221bcbce3663db245c0dd5f78bbd748650f3e0b0d"} Nov 02 13:50:29.814618 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:29.814119 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" event={"ID":"f3247735-5d35-43ac-a9cd-48c53d43ab19","Type":"ContainerStarted","Data":"ce54f010e566d1a82c68a56ee6734bd12f01ed50236d2dca58e8fde734b89bdf"} Nov 02 13:50:29.814618 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:29.814149 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:29.870761 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:29.870707 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" podStartSLOduration=1.870694061 podStartE2EDuration="1.870694061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:29.858517853 +0000 UTC m=+378.584139648" watchObservedRunningTime="2026-11-02 13:50:29.870694061 +0000 UTC m=+378.596315832" Nov 02 13:50:30.819645 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:30.819616 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tnmvk" Nov 02 13:50:30.909944 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:30.909911 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5"] Nov 02 13:50:30.910205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:30.910181 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" podUID="21a77acb-04be-4ff6-97ee-b31b18a0700e" containerName="discovery" containerID="cri-o://f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047" gracePeriod=30 Nov 02 13:50:31.073908 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:50:31.073817 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:50:31.153290 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.153268 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:50:31.202111 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.202077 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-cacerts\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-cacerts\") pod \"21a77acb-04be-4ff6-97ee-b31b18a0700e\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " Nov 02 13:50:31.202266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.202144 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-ca-configmap\") pod \"21a77acb-04be-4ff6-97ee-b31b18a0700e\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " Nov 02 13:50:31.202266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.202178 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-token\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-token\") pod \"21a77acb-04be-4ff6-97ee-b31b18a0700e\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " Nov 02 13:50:31.202266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.202210 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-dns-cert\") pod \"21a77acb-04be-4ff6-97ee-b31b18a0700e\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " Nov 02 13:50:31.202266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.202246 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a77acb-04be-4ff6-97ee-b31b18a0700e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a77acb-04be-4ff6-97ee-b31b18a0700e-local-certs\") pod \"21a77acb-04be-4ff6-97ee-b31b18a0700e\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " Nov 02 13:50:31.202480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.202310 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-kube-api-access-r5bgx\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-kube-api-access-r5bgx\") pod \"21a77acb-04be-4ff6-97ee-b31b18a0700e\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " Nov 02 13:50:31.202480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.202377 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-kubeconfig\") pod \"21a77acb-04be-4ff6-97ee-b31b18a0700e\" (UID: \"21a77acb-04be-4ff6-97ee-b31b18a0700e\") " Nov 02 13:50:31.202798 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.202745 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-ca-configmap" pod "21a77acb-04be-4ff6-97ee-b31b18a0700e" (UID: "21a77acb-04be-4ff6-97ee-b31b18a0700e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:50:31.205407 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.205332 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-token" pod "21a77acb-04be-4ff6-97ee-b31b18a0700e" (UID: "21a77acb-04be-4ff6-97ee-b31b18a0700e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:50:31.205513 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.205461 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a77acb-04be-4ff6-97ee-b31b18a0700e-local-certs" pod "21a77acb-04be-4ff6-97ee-b31b18a0700e" (UID: "21a77acb-04be-4ff6-97ee-b31b18a0700e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:50:31.205665 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.205644 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-kube-api-access-r5bgx" pod "21a77acb-04be-4ff6-97ee-b31b18a0700e" (UID: "21a77acb-04be-4ff6-97ee-b31b18a0700e"). InnerVolumeSpecName "kube-api-access-r5bgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:50:31.206169 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.206145 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:31.206532 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.206510 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-dns-cert" pod "21a77acb-04be-4ff6-97ee-b31b18a0700e" (UID: "21a77acb-04be-4ff6-97ee-b31b18a0700e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:50:31.206820 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.206800 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-cacerts" pod "21a77acb-04be-4ff6-97ee-b31b18a0700e" (UID: "21a77acb-04be-4ff6-97ee-b31b18a0700e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:50:31.206971 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.206950 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-kubeconfig" pod "21a77acb-04be-4ff6-97ee-b31b18a0700e" (UID: "21a77acb-04be-4ff6-97ee-b31b18a0700e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:50:31.303081 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.303048 5591 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-cacerts\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:31.303081 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.303075 5591 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-ca-configmap\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:31.303081 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.303085 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:31.303296 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.303095 5591 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-csr-dns-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:31.303296 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.303107 5591 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a77acb-04be-4ff6-97ee-b31b18a0700e-local-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:31.303296 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.303115 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5bgx\" (UniqueName: \"kubernetes.io/projected/21a77acb-04be-4ff6-97ee-b31b18a0700e-kube-api-access-r5bgx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:31.303296 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.303123 5591 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21a77acb-04be-4ff6-97ee-b31b18a0700e-istio-kubeconfig\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:31.366314 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.366212 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:31.529712 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.529680 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:31.694656 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.694619 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:31.822914 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.822882 5591 generic.go:356] "Generic (PLEG): container finished" podID="21a77acb-04be-4ff6-97ee-b31b18a0700e" containerID="f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047" exitCode=0 Nov 02 13:50:31.823342 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.822932 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" Nov 02 13:50:31.823342 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.822960 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" event={"ID":"21a77acb-04be-4ff6-97ee-b31b18a0700e","Type":"ContainerDied","Data":"f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047"} Nov 02 13:50:31.823342 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.823022 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5" event={"ID":"21a77acb-04be-4ff6-97ee-b31b18a0700e","Type":"ContainerDied","Data":"b8a37cf72498b0e15d72b23a02e789455b6b25e48ed56cb214c62f0fc5333369"} Nov 02 13:50:31.823342 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.823055 5591 scope.go:122] "RemoveContainer" containerID="f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047" Nov 02 13:50:31.830838 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.830823 5591 scope.go:122] "RemoveContainer" containerID="f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047" Nov 02 13:50:31.831139 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:50:31.831118 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047\": container with ID starting with f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047 not found: ID does not exist" containerID="f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047" Nov 02 13:50:31.831193 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.831148 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047"} err="failed to get container status \"f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047\": rpc error: code = NotFound desc = could not find container \"f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047\": container with ID starting with f31258d88664f46116ccdf6f9575b0a526c4fbd571cd1957225925d316225047 not found: ID does not exist" Nov 02 13:50:31.877020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.876974 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5"] Nov 02 13:50:31.888464 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:31.888436 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-65kv5"] Nov 02 13:50:33.427668 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:33.427637 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a77acb-04be-4ff6-97ee-b31b18a0700e" path="/var/lib/kubelet/pods/21a77acb-04be-4ff6-97ee-b31b18a0700e/volumes" Nov 02 13:50:33.787352 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:33.787263 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:35.046612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.046579 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-pl7vn"] Nov 02 13:50:35.047043 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.046772 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" podUID="8b854bc5-df65-4c58-a7bc-30540708f129" containerName="manager" containerID="cri-o://3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748" gracePeriod=30 Nov 02 13:50:35.280265 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.280239 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:35.339968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.339884 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-metrics-certs\") pod \"8b854bc5-df65-4c58-a7bc-30540708f129\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " Nov 02 13:50:35.339968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.339920 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b854bc5-df65-4c58-a7bc-30540708f129-kube-api-access-8ww8k\" (UniqueName: \"kubernetes.io/projected/8b854bc5-df65-4c58-a7bc-30540708f129-kube-api-access-8ww8k\") pod \"8b854bc5-df65-4c58-a7bc-30540708f129\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " Nov 02 13:50:35.340223 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.340017 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-webhook-cert\") pod \"8b854bc5-df65-4c58-a7bc-30540708f129\" (UID: \"8b854bc5-df65-4c58-a7bc-30540708f129\") " Nov 02 13:50:35.342175 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.342145 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-webhook-cert" pod "8b854bc5-df65-4c58-a7bc-30540708f129" (UID: "8b854bc5-df65-4c58-a7bc-30540708f129"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:50:35.342260 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.342183 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-metrics-certs" pod "8b854bc5-df65-4c58-a7bc-30540708f129" (UID: "8b854bc5-df65-4c58-a7bc-30540708f129"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:50:35.342260 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.342203 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b854bc5-df65-4c58-a7bc-30540708f129-kube-api-access-8ww8k" pod "8b854bc5-df65-4c58-a7bc-30540708f129" (UID: "8b854bc5-df65-4c58-a7bc-30540708f129"). InnerVolumeSpecName "kube-api-access-8ww8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:50:35.440743 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.440715 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:35.440743 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.440741 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ww8k\" (UniqueName: \"kubernetes.io/projected/8b854bc5-df65-4c58-a7bc-30540708f129-kube-api-access-8ww8k\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:35.440897 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.440751 5591 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b854bc5-df65-4c58-a7bc-30540708f129-webhook-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:50:35.840459 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.840419 5591 generic.go:356] "Generic (PLEG): container finished" podID="8b854bc5-df65-4c58-a7bc-30540708f129" containerID="3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748" exitCode=0 Nov 02 13:50:35.840722 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.840504 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" event={"ID":"8b854bc5-df65-4c58-a7bc-30540708f129","Type":"ContainerDied","Data":"3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748"} Nov 02 13:50:35.840843 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.840757 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" event={"ID":"8b854bc5-df65-4c58-a7bc-30540708f129","Type":"ContainerDied","Data":"38865d5fb978f1a148423a6e2f39524f4487fd90eabaa425fd8654e536b923a6"} Nov 02 13:50:35.840843 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.840792 5591 scope.go:122] "RemoveContainer" containerID="3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748" Nov 02 13:50:35.840965 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.840544 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8d8c8bb-pl7vn" Nov 02 13:50:35.850226 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.850209 5591 scope.go:122] "RemoveContainer" containerID="3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748" Nov 02 13:50:35.850453 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:50:35.850435 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748\": container with ID starting with 3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748 not found: ID does not exist" containerID="3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748" Nov 02 13:50:35.850511 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.850461 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748"} err="failed to get container status \"3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748\": rpc error: code = NotFound desc = could not find container \"3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748\": container with ID starting with 3a49332684cde836b977ac627013c9e993ace53b531e44f67cd7e41d2cb28748 not found: ID does not exist" Nov 02 13:50:35.891871 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.891840 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-pl7vn"] Nov 02 13:50:35.903700 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:35.903675 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c8d8c8bb-pl7vn"] Nov 02 13:50:37.427750 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:37.427717 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b854bc5-df65-4c58-a7bc-30540708f129" path="/var/lib/kubelet/pods/8b854bc5-df65-4c58-a7bc-30540708f129/volumes" Nov 02 13:50:53.865056 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:53.865017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:54.048378 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:54.048344 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:55.275259 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.275225 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:55.504127 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.504083 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:55.793367 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.793327 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-7frbv"] Nov 02 13:50:55.793731 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.793712 5591 cpu_manager.go:414] "containerMap: removing container" podUID="21a77acb-04be-4ff6-97ee-b31b18a0700e" containerName="discovery" Nov 02 13:50:55.793818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.793734 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a77acb-04be-4ff6-97ee-b31b18a0700e" containerName="discovery" Nov 02 13:50:55.793818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.793758 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8b854bc5-df65-4c58-a7bc-30540708f129" containerName="manager" Nov 02 13:50:55.793818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.793766 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b854bc5-df65-4c58-a7bc-30540708f129" containerName="manager" Nov 02 13:50:55.793968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.793839 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a77acb-04be-4ff6-97ee-b31b18a0700e" containerName="discovery" Nov 02 13:50:55.793968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.793854 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b854bc5-df65-4c58-a7bc-30540708f129" containerName="manager" Nov 02 13:50:55.801015 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.800971 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:55.830050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.830022 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 13:50:55.830050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.830037 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 13:50:55.830050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.830041 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 13:50:55.830271 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.830032 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 13:50:55.830271 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.830180 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 13:50:55.830942 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.830923 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-7frbv"] Nov 02 13:50:55.913973 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.913938 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-maas-api-tls\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:55.914199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.914013 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-metrics-certs\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:55.914199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.914088 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c356f54e-53eb-4563-9bf1-eef603056f0a-openshift-service-ca\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:55.914199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:55.914125 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnhq\" (UniqueName: \"kubernetes.io/projected/c356f54e-53eb-4563-9bf1-eef603056f0a-kube-api-access-vbnhq\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.015506 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.015468 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnhq\" (UniqueName: \"kubernetes.io/projected/c356f54e-53eb-4563-9bf1-eef603056f0a-kube-api-access-vbnhq\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.015692 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.015515 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-maas-api-tls\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.015692 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.015564 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-metrics-certs\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.015692 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.015592 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c356f54e-53eb-4563-9bf1-eef603056f0a-openshift-service-ca\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.016236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.016205 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c356f54e-53eb-4563-9bf1-eef603056f0a-openshift-service-ca\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.018020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.017978 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-metrics-certs\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.018138 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.018090 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-maas-api-tls\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.039103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.039074 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnhq\" (UniqueName: \"kubernetes.io/projected/c356f54e-53eb-4563-9bf1-eef603056f0a-kube-api-access-vbnhq\") pod \"maas-api-777f545dbb-7frbv\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.113680 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.113596 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:56.409962 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.409934 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-7frbv"] Nov 02 13:50:56.413139 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:50:56.413104 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc356f54e_53eb_4563_9bf1_eef603056f0a.slice/crio-3cc4ef700eff28d42b87a2270ec439bba7ff7ad9df5d6bd7fc1a4e6e92172592 WatchSource:0}: Error finding container 3cc4ef700eff28d42b87a2270ec439bba7ff7ad9df5d6bd7fc1a4e6e92172592: Status 404 returned error can't find the container with id 3cc4ef700eff28d42b87a2270ec439bba7ff7ad9df5d6bd7fc1a4e6e92172592 Nov 02 13:50:56.922339 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:56.922298 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" event={"ID":"c356f54e-53eb-4563-9bf1-eef603056f0a","Type":"ContainerStarted","Data":"3cc4ef700eff28d42b87a2270ec439bba7ff7ad9df5d6bd7fc1a4e6e92172592"} Nov 02 13:50:57.327013 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:57.326912 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:50:57.927307 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:57.927270 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" event={"ID":"c356f54e-53eb-4563-9bf1-eef603056f0a","Type":"ContainerStarted","Data":"b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628"} Nov 02 13:50:57.927806 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:57.927430 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:50:57.972992 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:50:57.972942 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" podStartSLOduration=1.6161384349999999 podStartE2EDuration="2.972927419s" totalImagesPullingTime="1.356788984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:57.972043699 +0000 UTC m=+406.697665469" watchObservedRunningTime="2026-11-02 13:50:57.972927419 +0000 UTC m=+406.698549183" Nov 02 13:51:01.072542 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:51:01.072500 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:51:03.937367 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:03.937338 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:51:04.459426 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:04.459398 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:04.843125 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:04.843093 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:14.331737 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.331704 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-7frbv"] Nov 02 13:51:14.332253 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.331916 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" podUID="c356f54e-53eb-4563-9bf1-eef603056f0a" containerName="maas-api" containerID="cri-o://b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628" gracePeriod=30 Nov 02 13:51:14.568280 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.568255 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:51:14.576492 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.576470 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-maas-api-tls\") pod \"c356f54e-53eb-4563-9bf1-eef603056f0a\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " Nov 02 13:51:14.576550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.576509 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c356f54e-53eb-4563-9bf1-eef603056f0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c356f54e-53eb-4563-9bf1-eef603056f0a-openshift-service-ca\") pod \"c356f54e-53eb-4563-9bf1-eef603056f0a\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " Nov 02 13:51:14.576632 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.576613 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-metrics-certs\") pod \"c356f54e-53eb-4563-9bf1-eef603056f0a\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " Nov 02 13:51:14.576894 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.576870 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c356f54e-53eb-4563-9bf1-eef603056f0a-openshift-service-ca" pod "c356f54e-53eb-4563-9bf1-eef603056f0a" (UID: "c356f54e-53eb-4563-9bf1-eef603056f0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:51:14.578641 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.578618 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-maas-api-tls" pod "c356f54e-53eb-4563-9bf1-eef603056f0a" (UID: "c356f54e-53eb-4563-9bf1-eef603056f0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:51:14.578754 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.578715 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-metrics-certs" pod "c356f54e-53eb-4563-9bf1-eef603056f0a" (UID: "c356f54e-53eb-4563-9bf1-eef603056f0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:51:14.677854 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.677818 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c356f54e-53eb-4563-9bf1-eef603056f0a-kube-api-access-vbnhq\" (UniqueName: \"kubernetes.io/projected/c356f54e-53eb-4563-9bf1-eef603056f0a-kube-api-access-vbnhq\") pod \"c356f54e-53eb-4563-9bf1-eef603056f0a\" (UID: \"c356f54e-53eb-4563-9bf1-eef603056f0a\") " Nov 02 13:51:14.678069 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.678003 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:51:14.678069 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.678023 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c356f54e-53eb-4563-9bf1-eef603056f0a-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:51:14.678069 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.678033 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c356f54e-53eb-4563-9bf1-eef603056f0a-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:51:14.679847 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.679823 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c356f54e-53eb-4563-9bf1-eef603056f0a-kube-api-access-vbnhq" pod "c356f54e-53eb-4563-9bf1-eef603056f0a" (UID: "c356f54e-53eb-4563-9bf1-eef603056f0a"). InnerVolumeSpecName "kube-api-access-vbnhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:51:14.779100 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.779066 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbnhq\" (UniqueName: \"kubernetes.io/projected/c356f54e-53eb-4563-9bf1-eef603056f0a-kube-api-access-vbnhq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:51:14.993817 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.993780 5591 generic.go:356] "Generic (PLEG): container finished" podID="c356f54e-53eb-4563-9bf1-eef603056f0a" containerID="b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628" exitCode=0 Nov 02 13:51:14.994026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.993824 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" Nov 02 13:51:14.994026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.993866 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" event={"ID":"c356f54e-53eb-4563-9bf1-eef603056f0a","Type":"ContainerDied","Data":"b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628"} Nov 02 13:51:14.994026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.993919 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777f545dbb-7frbv" event={"ID":"c356f54e-53eb-4563-9bf1-eef603056f0a","Type":"ContainerDied","Data":"3cc4ef700eff28d42b87a2270ec439bba7ff7ad9df5d6bd7fc1a4e6e92172592"} Nov 02 13:51:14.994026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:14.993942 5591 scope.go:122] "RemoveContainer" containerID="b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628" Nov 02 13:51:15.002561 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:15.002541 5591 scope.go:122] "RemoveContainer" containerID="b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628" Nov 02 13:51:15.002832 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:51:15.002811 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628\": container with ID starting with b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628 not found: ID does not exist" containerID="b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628" Nov 02 13:51:15.002882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:15.002844 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628"} err="failed to get container status \"b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628\": rpc error: code = NotFound desc = could not find container \"b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628\": container with ID starting with b0f29c571e465e508fee71245a2e916ec01acb0fd09138c916f11c70b0579628 not found: ID does not exist" Nov 02 13:51:15.044533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:15.044494 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-7frbv"] Nov 02 13:51:15.056562 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:15.056523 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-777f545dbb-7frbv"] Nov 02 13:51:15.427526 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:15.427497 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c356f54e-53eb-4563-9bf1-eef603056f0a" path="/var/lib/kubelet/pods/c356f54e-53eb-4563-9bf1-eef603056f0a/volumes" Nov 02 13:51:20.024619 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:20.024579 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:23.072604 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:23.072557 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:23.829111 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:23.829071 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:23.989118 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:51:23.989085 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:31.072951 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:51:31.072914 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:52:01.072846 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:52:01.072754 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:52:01.592545 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.592499 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-fv45g"] Nov 02 13:52:01.592890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.592870 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c356f54e-53eb-4563-9bf1-eef603056f0a" containerName="maas-api" Nov 02 13:52:01.592966 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.592891 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c356f54e-53eb-4563-9bf1-eef603056f0a" containerName="maas-api" Nov 02 13:52:01.593047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.593034 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c356f54e-53eb-4563-9bf1-eef603056f0a" containerName="maas-api" Nov 02 13:52:01.594849 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.594828 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:01.621307 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.621274 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-fv45g"] Nov 02 13:52:01.624628 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.624595 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 13:52:01.624773 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.624756 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 13:52:01.624827 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.624783 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-pq5qz\"" Nov 02 13:52:01.624883 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.624818 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 13:52:01.624883 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.624846 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 13:52:01.630465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.630422 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:01.749193 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.749153 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d81c802-6813-4fe5-910d-9e42e460c72d-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-fv45g\" (UID: \"0d81c802-6813-4fe5-910d-9e42e460c72d\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:01.749368 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.749228 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlvg\" (UniqueName: \"kubernetes.io/projected/0d81c802-6813-4fe5-910d-9e42e460c72d-kube-api-access-hjlvg\") pod \"ai-gateway-controller-5d8dddb676-fv45g\" (UID: \"0d81c802-6813-4fe5-910d-9e42e460c72d\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:01.831891 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.831847 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:01.850305 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.850241 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlvg\" (UniqueName: \"kubernetes.io/projected/0d81c802-6813-4fe5-910d-9e42e460c72d-kube-api-access-hjlvg\") pod \"ai-gateway-controller-5d8dddb676-fv45g\" (UID: \"0d81c802-6813-4fe5-910d-9e42e460c72d\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:01.850305 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.850293 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d81c802-6813-4fe5-910d-9e42e460c72d-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-fv45g\" (UID: \"0d81c802-6813-4fe5-910d-9e42e460c72d\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:01.852603 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.852583 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d81c802-6813-4fe5-910d-9e42e460c72d-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-fv45g\" (UID: \"0d81c802-6813-4fe5-910d-9e42e460c72d\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:01.873608 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.873584 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlvg\" (UniqueName: \"kubernetes.io/projected/0d81c802-6813-4fe5-910d-9e42e460c72d-kube-api-access-hjlvg\") pod \"ai-gateway-controller-5d8dddb676-fv45g\" (UID: \"0d81c802-6813-4fe5-910d-9e42e460c72d\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:01.906080 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:01.906050 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:02.201711 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:02.201673 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-fv45g"] Nov 02 13:52:02.205218 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:52:02.205180 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d81c802_6813_4fe5_910d_9e42e460c72d.slice/crio-d067e8b30de22549819b6e7329f1727405e0569e3d226a81128c8793cb18a46f WatchSource:0}: Error finding container d067e8b30de22549819b6e7329f1727405e0569e3d226a81128c8793cb18a46f: Status 404 returned error can't find the container with id d067e8b30de22549819b6e7329f1727405e0569e3d226a81128c8793cb18a46f Nov 02 13:52:03.177853 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:03.177817 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" event={"ID":"0d81c802-6813-4fe5-910d-9e42e460c72d","Type":"ContainerStarted","Data":"d067e8b30de22549819b6e7329f1727405e0569e3d226a81128c8793cb18a46f"} Nov 02 13:52:04.183316 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:04.183274 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" event={"ID":"0d81c802-6813-4fe5-910d-9e42e460c72d","Type":"ContainerStarted","Data":"3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae"} Nov 02 13:52:04.183824 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:04.183418 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:04.226667 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:04.226599 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" podStartSLOduration=1.726404885 podStartE2EDuration="3.226587264s" totalImagesPullingTime="1.500182379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:04.225779461 +0000 UTC m=+472.951401232" watchObservedRunningTime="2026-11-02 13:52:04.226587264 +0000 UTC m=+472.952209033" Nov 02 13:52:05.830077 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.830042 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:05.869074 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.869039 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-t792x"] Nov 02 13:52:05.871419 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.871397 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:05.889044 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.889020 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 13:52:05.898661 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.898638 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-t792x"] Nov 02 13:52:05.949186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.949153 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9"] Nov 02 13:52:05.951407 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.951386 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:05.967886 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.967863 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-964jt\"" Nov 02 13:52:05.973596 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.973571 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9"] Nov 02 13:52:05.986055 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.986028 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gr6\" (UniqueName: \"kubernetes.io/projected/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-kube-api-access-h5gr6\") pod \"payload-processing-55fbbbf9dc-t792x\" (UID: \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:05.986170 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.986060 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-t792x\" (UID: \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:05.999759 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:05.999731 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:06.087188 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.087108 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2wf\" (UniqueName: \"kubernetes.io/projected/91a47106-23e2-439f-abdb-622b606887c4-kube-api-access-zn2wf\") pod \"payload-pre-processing-7b98bb678c-cxzs9\" (UID: \"91a47106-23e2-439f-abdb-622b606887c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:06.087188 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.087167 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gr6\" (UniqueName: \"kubernetes.io/projected/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-kube-api-access-h5gr6\") pod \"payload-processing-55fbbbf9dc-t792x\" (UID: \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:06.087390 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.087239 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-t792x\" (UID: \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:06.087390 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.087267 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a47106-23e2-439f-abdb-622b606887c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cxzs9\" (UID: \"91a47106-23e2-439f-abdb-622b606887c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:06.087894 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.087878 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-t792x\" (UID: \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:06.110896 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.110866 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gr6\" (UniqueName: \"kubernetes.io/projected/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-kube-api-access-h5gr6\") pod \"payload-processing-55fbbbf9dc-t792x\" (UID: \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:06.182315 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.182271 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:06.188629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.188602 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2wf\" (UniqueName: \"kubernetes.io/projected/91a47106-23e2-439f-abdb-622b606887c4-kube-api-access-zn2wf\") pod \"payload-pre-processing-7b98bb678c-cxzs9\" (UID: \"91a47106-23e2-439f-abdb-622b606887c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:06.188767 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.188751 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a47106-23e2-439f-abdb-622b606887c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cxzs9\" (UID: \"91a47106-23e2-439f-abdb-622b606887c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:06.189491 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.189470 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a47106-23e2-439f-abdb-622b606887c4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-cxzs9\" (UID: \"91a47106-23e2-439f-abdb-622b606887c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:06.212732 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.212699 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2wf\" (UniqueName: \"kubernetes.io/projected/91a47106-23e2-439f-abdb-622b606887c4-kube-api-access-zn2wf\") pod \"payload-pre-processing-7b98bb678c-cxzs9\" (UID: \"91a47106-23e2-439f-abdb-622b606887c4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:06.262368 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.262340 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:06.476562 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.476453 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-t792x"] Nov 02 13:52:06.479027 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:52:06.479001 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566435d0_5ce9_4ef4_9a61_9ebb69b8d17d.slice/crio-3b6b5c7d12682f29109152c3176a03fc87913c28ced9df920491cdd504c5c4aa WatchSource:0}: Error finding container 3b6b5c7d12682f29109152c3176a03fc87913c28ced9df920491cdd504c5c4aa: Status 404 returned error can't find the container with id 3b6b5c7d12682f29109152c3176a03fc87913c28ced9df920491cdd504c5c4aa Nov 02 13:52:06.559079 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:06.559047 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9"] Nov 02 13:52:06.562126 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:52:06.562102 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a47106_23e2_439f_abdb_622b606887c4.slice/crio-def028f2b5fcd75814286526c7d57e9dec1f51626bd08a35e67bc253908c4201 WatchSource:0}: Error finding container def028f2b5fcd75814286526c7d57e9dec1f51626bd08a35e67bc253908c4201: Status 404 returned error can't find the container with id def028f2b5fcd75814286526c7d57e9dec1f51626bd08a35e67bc253908c4201 Nov 02 13:52:07.200331 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:07.200289 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" event={"ID":"91a47106-23e2-439f-abdb-622b606887c4","Type":"ContainerStarted","Data":"def028f2b5fcd75814286526c7d57e9dec1f51626bd08a35e67bc253908c4201"} Nov 02 13:52:07.202240 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:07.202150 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" event={"ID":"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d","Type":"ContainerStarted","Data":"3b6b5c7d12682f29109152c3176a03fc87913c28ced9df920491cdd504c5c4aa"} Nov 02 13:52:09.213690 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:09.213589 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" event={"ID":"91a47106-23e2-439f-abdb-622b606887c4","Type":"ContainerStarted","Data":"43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6"} Nov 02 13:52:09.214132 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:09.213696 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:09.214929 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:09.214910 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" event={"ID":"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d","Type":"ContainerStarted","Data":"9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d"} Nov 02 13:52:09.215049 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:09.215034 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:09.254600 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:09.254548 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" podStartSLOduration=1.867917922 podStartE2EDuration="4.254535566s" totalImagesPullingTime="2.386617644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:09.253785569 +0000 UTC m=+477.979407336" watchObservedRunningTime="2026-11-02 13:52:09.254535566 +0000 UTC m=+477.980157335" Nov 02 13:52:09.294024 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:09.293947 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" podStartSLOduration=1.860680496 podStartE2EDuration="4.293932991s" totalImagesPullingTime="2.433252495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:09.293429011 +0000 UTC m=+478.019050781" watchObservedRunningTime="2026-11-02 13:52:09.293932991 +0000 UTC m=+478.019554761" Nov 02 13:52:13.580391 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:13.580349 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:15.193701 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.193661 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:15.479266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.479187 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v"] Nov 02 13:52:15.483882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.483862 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:15.504450 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.504416 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v"] Nov 02 13:52:15.568140 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.568101 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmgd\" (UniqueName: \"kubernetes.io/projected/a658ad5a-9b96-42c1-af92-c22f12558516-kube-api-access-jvmgd\") pod \"ai-gateway-controller-7b7f5f45bd-8zr8v\" (UID: \"a658ad5a-9b96-42c1-af92-c22f12558516\") " pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:15.568140 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.568144 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a658ad5a-9b96-42c1-af92-c22f12558516-metrics-certs\") pod \"ai-gateway-controller-7b7f5f45bd-8zr8v\" (UID: \"a658ad5a-9b96-42c1-af92-c22f12558516\") " pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:15.668672 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.668606 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a658ad5a-9b96-42c1-af92-c22f12558516-metrics-certs\") pod \"ai-gateway-controller-7b7f5f45bd-8zr8v\" (UID: \"a658ad5a-9b96-42c1-af92-c22f12558516\") " pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:15.668837 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.668716 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmgd\" (UniqueName: \"kubernetes.io/projected/a658ad5a-9b96-42c1-af92-c22f12558516-kube-api-access-jvmgd\") pod \"ai-gateway-controller-7b7f5f45bd-8zr8v\" (UID: \"a658ad5a-9b96-42c1-af92-c22f12558516\") " pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:15.671086 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.671062 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a658ad5a-9b96-42c1-af92-c22f12558516-metrics-certs\") pod \"ai-gateway-controller-7b7f5f45bd-8zr8v\" (UID: \"a658ad5a-9b96-42c1-af92-c22f12558516\") " pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:15.692651 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.692619 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmgd\" (UniqueName: \"kubernetes.io/projected/a658ad5a-9b96-42c1-af92-c22f12558516-kube-api-access-jvmgd\") pod \"ai-gateway-controller-7b7f5f45bd-8zr8v\" (UID: \"a658ad5a-9b96-42c1-af92-c22f12558516\") " pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:15.796471 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:15.796363 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:16.092687 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:16.092603 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v"] Nov 02 13:52:16.096119 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:52:16.096080 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda658ad5a_9b96_42c1_af92_c22f12558516.slice/crio-e1c42be4217988006d9cc655bd98c820167d76e4d49d1388ae867b34eab2ad85 WatchSource:0}: Error finding container e1c42be4217988006d9cc655bd98c820167d76e4d49d1388ae867b34eab2ad85: Status 404 returned error can't find the container with id e1c42be4217988006d9cc655bd98c820167d76e4d49d1388ae867b34eab2ad85 Nov 02 13:52:16.242078 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:16.242041 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" event={"ID":"a658ad5a-9b96-42c1-af92-c22f12558516","Type":"ContainerStarted","Data":"e1c42be4217988006d9cc655bd98c820167d76e4d49d1388ae867b34eab2ad85"} Nov 02 13:52:17.247533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:17.247494 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" event={"ID":"a658ad5a-9b96-42c1-af92-c22f12558516","Type":"ContainerStarted","Data":"7db1434bbc2e5ee5a86fd864ebccf41376256f899cbafa97d40a6d9087cb491e"} Nov 02 13:52:17.248036 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:17.247667 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:17.289598 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:17.289543 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" podStartSLOduration=1.951587969 podStartE2EDuration="2.289527016s" totalImagesPullingTime="337.939047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:17.288580219 +0000 UTC m=+486.014201990" watchObservedRunningTime="2026-11-02 13:52:17.289527016 +0000 UTC m=+486.015148776" Nov 02 13:52:20.141789 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:20.141756 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:20.222260 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:20.222232 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 13:52:20.222401 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:20.222300 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 13:52:28.253902 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.253871 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7b7f5f45bd-8zr8v" Nov 02 13:52:28.320565 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.320527 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-fv45g"] Nov 02 13:52:28.320768 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.320743 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" podUID="0d81c802-6813-4fe5-910d-9e42e460c72d" containerName="manager" containerID="cri-o://3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae" gracePeriod=30 Nov 02 13:52:28.564694 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.564667 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:28.674921 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.674887 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d81c802-6813-4fe5-910d-9e42e460c72d-kube-api-access-hjlvg\" (UniqueName: \"kubernetes.io/projected/0d81c802-6813-4fe5-910d-9e42e460c72d-kube-api-access-hjlvg\") pod \"0d81c802-6813-4fe5-910d-9e42e460c72d\" (UID: \"0d81c802-6813-4fe5-910d-9e42e460c72d\") " Nov 02 13:52:28.675096 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.674929 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d81c802-6813-4fe5-910d-9e42e460c72d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d81c802-6813-4fe5-910d-9e42e460c72d-metrics-certs\") pod \"0d81c802-6813-4fe5-910d-9e42e460c72d\" (UID: \"0d81c802-6813-4fe5-910d-9e42e460c72d\") " Nov 02 13:52:28.677166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.677136 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d81c802-6813-4fe5-910d-9e42e460c72d-kube-api-access-hjlvg" pod "0d81c802-6813-4fe5-910d-9e42e460c72d" (UID: "0d81c802-6813-4fe5-910d-9e42e460c72d"). InnerVolumeSpecName "kube-api-access-hjlvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:52:28.677280 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.677137 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d81c802-6813-4fe5-910d-9e42e460c72d-metrics-certs" pod "0d81c802-6813-4fe5-910d-9e42e460c72d" (UID: "0d81c802-6813-4fe5-910d-9e42e460c72d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:52:28.776480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.776430 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjlvg\" (UniqueName: \"kubernetes.io/projected/0d81c802-6813-4fe5-910d-9e42e460c72d-kube-api-access-hjlvg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:52:28.776480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:28.776474 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d81c802-6813-4fe5-910d-9e42e460c72d-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:52:29.297486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.297452 5591 generic.go:356] "Generic (PLEG): container finished" podID="0d81c802-6813-4fe5-910d-9e42e460c72d" containerID="3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae" exitCode=0 Nov 02 13:52:29.297910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.297498 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" Nov 02 13:52:29.297910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.297513 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" event={"ID":"0d81c802-6813-4fe5-910d-9e42e460c72d","Type":"ContainerDied","Data":"3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae"} Nov 02 13:52:29.297910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.297555 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-fv45g" event={"ID":"0d81c802-6813-4fe5-910d-9e42e460c72d","Type":"ContainerDied","Data":"d067e8b30de22549819b6e7329f1727405e0569e3d226a81128c8793cb18a46f"} Nov 02 13:52:29.297910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.297576 5591 scope.go:122] "RemoveContainer" containerID="3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae" Nov 02 13:52:29.311474 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.311454 5591 scope.go:122] "RemoveContainer" containerID="3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae" Nov 02 13:52:29.311762 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:52:29.311742 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae\": container with ID starting with 3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae not found: ID does not exist" containerID="3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae" Nov 02 13:52:29.311831 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.311776 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae"} err="failed to get container status \"3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae\": rpc error: code = NotFound desc = could not find container \"3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae\": container with ID starting with 3a86252628dc1968c2402ed9a50cbb15e3e7188541b495b7387f83e678a3abae not found: ID does not exist" Nov 02 13:52:29.349629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.349594 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-fv45g"] Nov 02 13:52:29.361778 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.361744 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-fv45g"] Nov 02 13:52:29.427644 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:29.427610 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d81c802-6813-4fe5-910d-9e42e460c72d" path="/var/lib/kubelet/pods/0d81c802-6813-4fe5-910d-9e42e460c72d/volumes" Nov 02 13:52:30.006369 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:30.006333 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:30.517157 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:30.517120 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:31.073260 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:52:31.073225 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:52:31.713415 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:31.713382 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:31.884568 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:31.884532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:35.225347 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.225312 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2"] Nov 02 13:52:35.225880 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.225769 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0d81c802-6813-4fe5-910d-9e42e460c72d" containerName="manager" Nov 02 13:52:35.225880 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.225788 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d81c802-6813-4fe5-910d-9e42e460c72d" containerName="manager" Nov 02 13:52:35.225961 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.225890 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d81c802-6813-4fe5-910d-9e42e460c72d" containerName="manager" Nov 02 13:52:35.230522 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.230500 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.246333 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.246310 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vn5j4\"" Nov 02 13:52:35.258553 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.258524 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 13:52:35.258553 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.258525 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 13:52:35.258766 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.258560 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 13:52:35.261425 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.261396 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2"] Nov 02 13:52:35.330796 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.330763 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ac7768-2c53-40ab-af16-1d08fa52e223-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.330976 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.330804 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.330976 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.330834 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4q5\" (UniqueName: \"kubernetes.io/projected/13ac7768-2c53-40ab-af16-1d08fa52e223-kube-api-access-tj4q5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.330976 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.330946 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.331123 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.331002 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.331123 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.331047 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.358817 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.358783 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:35.431873 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.431839 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4q5\" (UniqueName: \"kubernetes.io/projected/13ac7768-2c53-40ab-af16-1d08fa52e223-kube-api-access-tj4q5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.431873 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.431874 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.432103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.432014 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.432103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.432091 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.432179 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.432120 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ac7768-2c53-40ab-af16-1d08fa52e223-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.432179 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.432153 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.432296 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.432274 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.432358 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.432338 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.432475 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.432458 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.434358 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.434332 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ac7768-2c53-40ab-af16-1d08fa52e223-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.434490 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.434471 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ac7768-2c53-40ab-af16-1d08fa52e223-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.455280 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.455241 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4q5\" (UniqueName: \"kubernetes.io/projected/13ac7768-2c53-40ab-af16-1d08fa52e223-kube-api-access-tj4q5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2\" (UID: \"13ac7768-2c53-40ab-af16-1d08fa52e223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.544494 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.544410 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:35.842797 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:35.842718 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2"] Nov 02 13:52:35.845673 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:52:35.845641 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ac7768_2c53_40ab_af16_1d08fa52e223.slice/crio-068392078d645851dcfa75135bbaacefb70dc2b499fea5bf08ed9dc1b25064d2 WatchSource:0}: Error finding container 068392078d645851dcfa75135bbaacefb70dc2b499fea5bf08ed9dc1b25064d2: Status 404 returned error can't find the container with id 068392078d645851dcfa75135bbaacefb70dc2b499fea5bf08ed9dc1b25064d2 Nov 02 13:52:36.331224 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:36.331168 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" event={"ID":"13ac7768-2c53-40ab-af16-1d08fa52e223","Type":"ContainerStarted","Data":"068392078d645851dcfa75135bbaacefb70dc2b499fea5bf08ed9dc1b25064d2"} Nov 02 13:52:36.362708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:36.362663 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:52:37.336387 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:37.336297 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" event={"ID":"13ac7768-2c53-40ab-af16-1d08fa52e223","Type":"ContainerStarted","Data":"a175dfa6c86667c1c8a54bd0b2112d131f6b62dec0b9e7d739529ee11cee305e"} Nov 02 13:52:37.381487 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:37.381430 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" podStartSLOduration=1.328806131 podStartE2EDuration="2.381415426s" totalImagesPullingTime="1.052609295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:37.380740083 +0000 UTC m=+506.106361854" watchObservedRunningTime="2026-11-02 13:52:37.381415426 +0000 UTC m=+506.107037214" Nov 02 13:52:37.545211 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:37.545177 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:40.356537 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:40.356497 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:41.447658 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.447624 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm"] Nov 02 13:52:41.453086 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.453067 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.485796 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.485761 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm"] Nov 02 13:52:41.487421 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.487386 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:52:41.488177 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.488158 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 13:52:41.583099 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.583061 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.583099 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.583102 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/597f5679-40fd-4228-9182-86f23da49976-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.583299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.583184 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.583299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.583221 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4jf\" (UniqueName: \"kubernetes.io/projected/597f5679-40fd-4228-9182-86f23da49976-kube-api-access-hn4jf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.583299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.583276 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.583397 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.583367 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.602571 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.602519 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:41.683777 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.683746 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4jf\" (UniqueName: \"kubernetes.io/projected/597f5679-40fd-4228-9182-86f23da49976-kube-api-access-hn4jf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.683959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.683798 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.683959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.683842 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.683959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.683886 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.683959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.683917 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/597f5679-40fd-4228-9182-86f23da49976-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.683959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.683958 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.684266 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.684242 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.684358 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.684338 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.684423 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.684405 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.686549 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.686528 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/597f5679-40fd-4228-9182-86f23da49976-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.686888 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.686863 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/597f5679-40fd-4228-9182-86f23da49976-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.707798 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.707725 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4jf\" (UniqueName: \"kubernetes.io/projected/597f5679-40fd-4228-9182-86f23da49976-kube-api-access-hn4jf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xppxm\" (UID: \"597f5679-40fd-4228-9182-86f23da49976\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:41.766907 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:41.766871 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:42.064248 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:42.064175 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm"] Nov 02 13:52:42.067379 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:52:42.067350 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597f5679_40fd_4228_9182_86f23da49976.slice/crio-d3129c32f690d7b0e59dcb89774207b2df8556eaae9add3f8857612f10e7f6cb WatchSource:0}: Error finding container d3129c32f690d7b0e59dcb89774207b2df8556eaae9add3f8857612f10e7f6cb: Status 404 returned error can't find the container with id d3129c32f690d7b0e59dcb89774207b2df8556eaae9add3f8857612f10e7f6cb Nov 02 13:52:42.357837 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:42.357803 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" event={"ID":"597f5679-40fd-4228-9182-86f23da49976","Type":"ContainerStarted","Data":"3aaea6cfd2c39d975c9febc4a49772368035d930e5a65d5a6e4813845ee99093"} Nov 02 13:52:42.358014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:42.357854 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" event={"ID":"597f5679-40fd-4228-9182-86f23da49976","Type":"ContainerStarted","Data":"d3129c32f690d7b0e59dcb89774207b2df8556eaae9add3f8857612f10e7f6cb"} Nov 02 13:52:42.405032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:42.404957 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" podStartSLOduration=1.207041253 podStartE2EDuration="1.404945125s" totalImagesPullingTime="197.903872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:42.403897852 +0000 UTC m=+511.129519621" watchObservedRunningTime="2026-11-02 13:52:42.404945125 +0000 UTC m=+511.130566894" Nov 02 13:52:42.765960 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:42.765923 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:42.767772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:42.767743 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:44.524533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.524500 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6"] Nov 02 13:52:44.531869 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.531839 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.558506 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.558470 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 13:52:44.559546 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.559523 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6"] Nov 02 13:52:44.610634 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.610597 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc319d8-bcb7-4352-b711-f72f71b56811-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.610634 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.610635 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.610848 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.610656 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.610848 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.610755 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mj5g\" (UniqueName: \"kubernetes.io/projected/5fc319d8-bcb7-4352-b711-f72f71b56811-kube-api-access-6mj5g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.610848 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.610795 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.611028 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.610876 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.641717 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.641685 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:44.712015 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.711969 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.712175 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.712046 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc319d8-bcb7-4352-b711-f72f71b56811-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.712175 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.712073 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.712175 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.712099 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.712175 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.712131 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mj5g\" (UniqueName: \"kubernetes.io/projected/5fc319d8-bcb7-4352-b711-f72f71b56811-kube-api-access-6mj5g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.712408 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.712180 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.712536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.712510 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.712604 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.712531 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.712604 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.712559 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.714335 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.714314 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc319d8-bcb7-4352-b711-f72f71b56811-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.714508 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.714491 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc319d8-bcb7-4352-b711-f72f71b56811-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.738388 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.738356 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mj5g\" (UniqueName: \"kubernetes.io/projected/5fc319d8-bcb7-4352-b711-f72f71b56811-kube-api-access-6mj5g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6\" (UID: \"5fc319d8-bcb7-4352-b711-f72f71b56811\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:44.845594 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:44.845505 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:45.144840 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.144796 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6"] Nov 02 13:52:45.152214 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:52:45.152159 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc319d8_bcb7_4352_b711_f72f71b56811.slice/crio-58c1ff2c468177ab72cc19fc395a1d21ab24acdb6435f8c87adfdc998354867b WatchSource:0}: Error finding container 58c1ff2c468177ab72cc19fc395a1d21ab24acdb6435f8c87adfdc998354867b: Status 404 returned error can't find the container with id 58c1ff2c468177ab72cc19fc395a1d21ab24acdb6435f8c87adfdc998354867b Nov 02 13:52:45.371787 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.371753 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" event={"ID":"5fc319d8-bcb7-4352-b711-f72f71b56811","Type":"ContainerStarted","Data":"58c1ff2c468177ab72cc19fc395a1d21ab24acdb6435f8c87adfdc998354867b"} Nov 02 13:52:45.517791 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.517754 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6ccj9"] Nov 02 13:52:45.521306 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.521282 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.536598 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.536571 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 13:52:45.545157 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.545130 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:45.549315 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.549290 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6ccj9"] Nov 02 13:52:45.558094 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.558073 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:45.620470 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.620437 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.620653 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.620481 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvgz\" (UniqueName: \"kubernetes.io/projected/702ef238-5f96-4a86-b35e-9002f0738f78-kube-api-access-zmvgz\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.620653 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.620515 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-home\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.620653 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.620632 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.620788 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.620670 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.620860 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.620840 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702ef238-5f96-4a86-b35e-9002f0738f78-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.628742 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.628718 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:45.721802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.721725 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.721802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.721767 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvgz\" (UniqueName: \"kubernetes.io/projected/702ef238-5f96-4a86-b35e-9002f0738f78-kube-api-access-zmvgz\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.721802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.721793 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-home\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.722060 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.721829 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.722060 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.721855 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.722060 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.721888 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702ef238-5f96-4a86-b35e-9002f0738f78-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.722207 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.722176 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.722267 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.722241 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-home\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.722375 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.722356 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.724185 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.724157 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702ef238-5f96-4a86-b35e-9002f0738f78-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.724467 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.724446 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702ef238-5f96-4a86-b35e-9002f0738f78-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.744806 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.744777 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvgz\" (UniqueName: \"kubernetes.io/projected/702ef238-5f96-4a86-b35e-9002f0738f78-kube-api-access-zmvgz\") pod \"echo-simulator-kserve-64b6f8f96c-6ccj9\" (UID: \"702ef238-5f96-4a86-b35e-9002f0738f78\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:45.834758 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:45.834719 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:46.132114 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:46.131832 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6ccj9"] Nov 02 13:52:46.134422 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:52:46.134393 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702ef238_5f96_4a86_b35e_9002f0738f78.slice/crio-fbb12f4dca4ea00e27f2002f3d9588be95d68d34ae59255cb129ff2e38a32288 WatchSource:0}: Error finding container fbb12f4dca4ea00e27f2002f3d9588be95d68d34ae59255cb129ff2e38a32288: Status 404 returned error can't find the container with id fbb12f4dca4ea00e27f2002f3d9588be95d68d34ae59255cb129ff2e38a32288 Nov 02 13:52:46.377456 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:46.377423 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" event={"ID":"702ef238-5f96-4a86-b35e-9002f0738f78","Type":"ContainerStarted","Data":"fbb12f4dca4ea00e27f2002f3d9588be95d68d34ae59255cb129ff2e38a32288"} Nov 02 13:52:46.378965 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:46.378940 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" event={"ID":"5fc319d8-bcb7-4352-b711-f72f71b56811","Type":"ContainerStarted","Data":"6cec298da970fc74bd4a6addd7cbf625ef7e584cedb10d2b0f7287d51a6737c0"} Nov 02 13:52:46.390508 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:46.390455 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2" Nov 02 13:52:46.421198 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:46.421152 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" podStartSLOduration=2.227274737 podStartE2EDuration="2.421138523s" totalImagesPullingTime="193.863786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:46.420244425 +0000 UTC m=+515.145866195" watchObservedRunningTime="2026-11-02 13:52:46.421138523 +0000 UTC m=+515.146760292" Nov 02 13:52:46.664305 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:46.664228 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:46.846310 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:46.846272 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:47.743617 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:47.743590 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:52:48.390313 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:48.390273 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" event={"ID":"702ef238-5f96-4a86-b35e-9002f0738f78","Type":"ContainerStarted","Data":"c4f5462d82c86dbd068d6559f3e3fe5ee460bdae05d6bfc565d5512fb050d957"} Nov 02 13:52:48.436058 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:48.436010 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" podStartSLOduration=1.9154790990000001 podStartE2EDuration="3.435972784s" totalImagesPullingTime="1.520493685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:48.435251133 +0000 UTC m=+517.160872902" watchObservedRunningTime="2026-11-02 13:52:48.435972784 +0000 UTC m=+517.161594553" Nov 02 13:52:48.834932 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:48.834846 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:49.841774 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:49.841737 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:51.767344 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:51.767309 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:51.779748 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:51.779719 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:52.418894 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:52.418866 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xppxm" Nov 02 13:52:54.846656 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:54.846618 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:54.859140 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:54.859114 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:55.432653 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:55.432619 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6" Nov 02 13:52:55.835229 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:55.835140 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:55.847762 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:55.847732 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:56.435389 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:56.435358 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6ccj9" Nov 02 13:52:56.439365 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:52:56.439337 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:53:00.236531 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.236498 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7"] Nov 02 13:53:00.239836 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.239818 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.260480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.260456 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27gvt\"" Nov 02 13:53:00.271155 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.271126 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7"] Nov 02 13:53:00.352006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.351940 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.352199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.352017 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.352199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.352064 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.352199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.352155 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.352378 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.352247 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.352378 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.352289 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t45q\" (UniqueName: \"kubernetes.io/projected/dae59a45-652a-46dc-a5e7-c983d14b2b7e-kube-api-access-6t45q\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.352378 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.352347 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.352526 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.352437 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.352526 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.352490 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.453802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.453763 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.453802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.453803 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454071 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.453825 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454071 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.453948 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454071 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454008 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t45q\" (UniqueName: \"kubernetes.io/projected/dae59a45-652a-46dc-a5e7-c983d14b2b7e-kube-api-access-6t45q\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454071 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454068 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454247 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454137 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454247 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454202 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454247 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454238 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454280 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454313 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454544 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454514 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454679 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454617 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.454800 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.454777 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.457049 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.457022 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.457177 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.457159 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.477357 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.477325 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae59a45-652a-46dc-a5e7-c983d14b2b7e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.477486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.477448 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t45q\" (UniqueName: \"kubernetes.io/projected/dae59a45-652a-46dc-a5e7-c983d14b2b7e-kube-api-access-6t45q\") pod \"maas-default-gateway-openshift-default-6886b96f9-wrbs7\" (UID: \"dae59a45-652a-46dc-a5e7-c983d14b2b7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.553262 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.553170 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:00.869131 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:00.869060 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7"] Nov 02 13:53:00.871600 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:00.871568 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae59a45_652a_46dc_a5e7_c983d14b2b7e.slice/crio-dbbdbd59d7e4d1189bf6614dcc9ec93e39bcc4c1ca47bfe4d7282f2ed12f2315 WatchSource:0}: Error finding container dbbdbd59d7e4d1189bf6614dcc9ec93e39bcc4c1ca47bfe4d7282f2ed12f2315: Status 404 returned error can't find the container with id dbbdbd59d7e4d1189bf6614dcc9ec93e39bcc4c1ca47bfe4d7282f2ed12f2315 Nov 02 13:53:01.073335 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:53:01.073228 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:53:01.446432 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:01.446395 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" event={"ID":"dae59a45-652a-46dc-a5e7-c983d14b2b7e","Type":"ContainerStarted","Data":"dbbdbd59d7e4d1189bf6614dcc9ec93e39bcc4c1ca47bfe4d7282f2ed12f2315"} Nov 02 13:53:03.510600 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:03.510569 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:03.510872 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:03.510639 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:03.510872 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:03.510666 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:04.462596 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:04.462558 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" event={"ID":"dae59a45-652a-46dc-a5e7-c983d14b2b7e","Type":"ContainerStarted","Data":"f7f19e0a014d840172834e48af7462048b73d0444ed79ccd6e224a88743691de"} Nov 02 13:53:04.519642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:04.519584 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podStartSLOduration=1.8834779510000002 podStartE2EDuration="4.519570907s" totalImagesPullingTime="2.636092956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:04.518855788 +0000 UTC m=+533.244477558" watchObservedRunningTime="2026-11-02 13:53:04.519570907 +0000 UTC m=+533.245192676" Nov 02 13:53:04.556116 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:04.556054 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:04.556672 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:04.556640 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wrbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 02 13:53:04.556814 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:04.556705 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podUID="dae59a45-652a-46dc-a5e7-c983d14b2b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 02 13:53:05.554006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:05.553888 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wrbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 02 13:53:05.554006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:05.553948 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podUID="dae59a45-652a-46dc-a5e7-c983d14b2b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 02 13:53:06.556436 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:06.556397 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wrbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 02 13:53:06.556916 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:06.556462 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podUID="dae59a45-652a-46dc-a5e7-c983d14b2b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 02 13:53:07.556454 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:07.556415 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wrbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 02 13:53:07.556936 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:07.556475 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podUID="dae59a45-652a-46dc-a5e7-c983d14b2b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 02 13:53:08.556484 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:08.556444 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wrbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 02 13:53:08.557014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:08.556497 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podUID="dae59a45-652a-46dc-a5e7-c983d14b2b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 02 13:53:09.554623 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:09.554575 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wrbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 02 13:53:09.554819 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:09.554654 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podUID="dae59a45-652a-46dc-a5e7-c983d14b2b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 02 13:53:10.554437 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:10.554393 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wrbs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 02 13:53:10.554849 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:10.554463 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podUID="dae59a45-652a-46dc-a5e7-c983d14b2b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 02 13:53:11.823208 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:11.823112 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:11.823724 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:11.823699 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:11.824606 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:11.824585 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" Nov 02 13:53:12.570053 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:12.570019 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:14.152361 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:14.152326 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:15.301024 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.300338 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5"] Nov 02 13:53:15.308653 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.308625 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.338149 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.338114 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5"] Nov 02 13:53:15.412182 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412144 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.412182 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412184 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.412416 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412225 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.412416 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412328 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.412416 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412363 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.412416 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412386 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.412557 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412413 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlx9\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-kube-api-access-lxlx9\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.412557 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412473 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.412557 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.412506 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513349 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513314 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513349 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513353 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513382 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513435 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513463 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513486 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513506 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlx9\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-kube-api-access-lxlx9\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513544 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513601 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.513890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.513788 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.514135 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.514107 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.514231 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.514148 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.514231 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.514206 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.514299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.514206 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.515835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.515808 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.516148 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.516130 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.536863 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.536822 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlx9\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-kube-api-access-lxlx9\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.537221 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.537201 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-nvjl5\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.623228 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.623146 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:15.971043 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.971015 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5"] Nov 02 13:53:15.973470 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:15.973434 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8b9c04_2f85_4bed_96ce_7ea07e69317d.slice/crio-8f1c9cd451ee2279e0cbedbc704ea9e87817d29a57ea92bb16abaf85d052b052 WatchSource:0}: Error finding container 8f1c9cd451ee2279e0cbedbc704ea9e87817d29a57ea92bb16abaf85d052b052: Status 404 returned error can't find the container with id 8f1c9cd451ee2279e0cbedbc704ea9e87817d29a57ea92bb16abaf85d052b052 Nov 02 13:53:15.975800 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.975765 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:15.975894 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.975832 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:15.975894 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:15.975862 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:16.516577 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:16.516541 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" event={"ID":"8d8b9c04-2f85-4bed-96ce-7ea07e69317d","Type":"ContainerStarted","Data":"c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17"} Nov 02 13:53:16.516998 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:16.516594 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" event={"ID":"8d8b9c04-2f85-4bed-96ce-7ea07e69317d","Type":"ContainerStarted","Data":"8f1c9cd451ee2279e0cbedbc704ea9e87817d29a57ea92bb16abaf85d052b052"} Nov 02 13:53:16.612478 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:16.612413 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podStartSLOduration=1.6123976949999999 podStartE2EDuration="1.612397695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:16.611479513 +0000 UTC m=+545.337101280" watchObservedRunningTime="2026-11-02 13:53:16.612397695 +0000 UTC m=+545.338019464" Nov 02 13:53:16.623326 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:16.623289 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:17.625086 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:17.625053 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nvjl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 13:53:17.625512 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:17.625110 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 13:53:18.624178 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:18.624142 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nvjl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 13:53:18.624357 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:18.624199 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 13:53:19.623813 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:19.623768 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nvjl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 13:53:19.624342 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:19.623830 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 13:53:20.270290 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:20.270252 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:20.624138 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:20.624055 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nvjl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 13:53:20.624138 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:20.624129 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 13:53:21.624655 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:21.624614 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nvjl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 13:53:21.625101 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:21.624673 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 13:53:22.624091 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:22.624054 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nvjl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 13:53:22.624253 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:22.624117 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 13:53:24.278091 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:24.278057 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:24.278803 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:24.278782 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:24.279691 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:24.279667 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 13:53:30.353951 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.353918 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn"] Nov 02 13:53:30.357237 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.357219 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.386324 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.386290 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn"] Nov 02 13:53:30.467565 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467527 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cgq\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-kube-api-access-g2cgq\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.467565 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467566 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.467787 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467587 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.467787 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467655 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.467787 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467720 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.467787 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467747 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.467787 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467779 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.467942 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467852 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.467942 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.467871 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569288 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569252 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569288 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569291 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569307 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569360 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cgq\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-kube-api-access-g2cgq\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569379 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569399 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569423 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569473 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569529 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569887 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569723 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569951 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569890 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.569951 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.569925 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.570137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.570108 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.570210 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.570150 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.571861 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.571836 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.572096 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.572077 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.592441 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.592415 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.592651 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.592624 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cgq\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-kube-api-access-g2cgq\") pod \"maas-default-gateway-openshift-default-6886b96f9-7cjzn\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.670422 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.670385 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:30.994828 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:30.994750 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn"] Nov 02 13:53:30.998023 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:30.997994 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9def5c6_d319_49bd_9a73_3dfdfcd14843.slice/crio-569c6fd765f86242616ed3cbb17d4b50660bf6f7732acd2779b0224891733f89 WatchSource:0}: Error finding container 569c6fd765f86242616ed3cbb17d4b50660bf6f7732acd2779b0224891733f89: Status 404 returned error can't find the container with id 569c6fd765f86242616ed3cbb17d4b50660bf6f7732acd2779b0224891733f89 Nov 02 13:53:31.000157 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:31.000128 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:31.000222 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:31.000195 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:31.000258 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:31.000224 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:31.075920 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:53:31.075882 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:53:31.608686 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:31.608648 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" event={"ID":"b9def5c6-d319-49bd-9a73-3dfdfcd14843","Type":"ContainerStarted","Data":"16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158"} Nov 02 13:53:31.609189 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:31.608700 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" event={"ID":"b9def5c6-d319-49bd-9a73-3dfdfcd14843","Type":"ContainerStarted","Data":"569c6fd765f86242616ed3cbb17d4b50660bf6f7732acd2779b0224891733f89"} Nov 02 13:53:31.671319 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:31.671281 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:31.675615 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:31.675568 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podStartSLOduration=1.675557055 podStartE2EDuration="1.675557055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:31.674969785 +0000 UTC m=+560.400591554" watchObservedRunningTime="2026-11-02 13:53:31.675557055 +0000 UTC m=+560.401178825" Nov 02 13:53:32.671928 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:32.671886 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 02 13:53:32.672342 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:32.671944 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 02 13:53:33.671410 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:33.671378 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 02 13:53:33.671612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:33.671451 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 02 13:53:34.671425 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:34.671373 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 02 13:53:34.671837 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:34.671447 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 02 13:53:35.671668 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:35.671632 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 02 13:53:35.672071 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:35.671687 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 02 13:53:36.671116 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:36.671073 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 02 13:53:36.671271 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:36.671135 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 02 13:53:37.671258 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:37.671219 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Nov 02 13:53:37.671740 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:37.671276 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Nov 02 13:53:38.794149 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:38.794111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:38.964908 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:38.964859 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:39.143690 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.143657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:39.225719 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.225686 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:39.226131 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.226021 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:39.227041 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.227020 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 13:53:39.477212 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.477132 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:39.703091 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.703059 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:39.865186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.865101 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:39.921281 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.921246 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr"] Nov 02 13:53:39.932190 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.932159 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:39.950404 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.950373 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr"] Nov 02 13:53:39.959871 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:39.959834 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6232c163-gw-openshift-default-dockercfg-mjwfg\"" Nov 02 13:53:40.040126 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.039936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:40.061225 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061192 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-certs\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.061421 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061233 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-envoy\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.061421 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061256 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-token\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.061548 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061405 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-socket\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.061548 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061454 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-credential-socket\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.061548 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061518 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-data\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.061734 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061589 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istiod-ca-cert\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.061734 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061639 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8nd\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-kube-api-access-qx8nd\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.061734 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.061707 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-podinfo\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.162771 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.162733 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-podinfo\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.162973 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.162844 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-certs\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.162973 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.162872 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-envoy\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.162973 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.162898 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-token\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.162973 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.162930 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-socket\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.162973 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.162953 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-credential-socket\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.163244 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.163006 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-data\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.163244 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.163055 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istiod-ca-cert\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.163244 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.163083 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8nd\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-kube-api-access-qx8nd\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.163481 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.163453 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-credential-socket\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.163649 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.163495 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-socket\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.163749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.163524 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-certs\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.163749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.163652 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-data\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.163960 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.163943 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istiod-ca-cert\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.165187 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.165169 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-envoy\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.165505 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.165483 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-podinfo\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.191974 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.191935 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-token\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.192172 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.192132 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8nd\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-kube-api-access-qx8nd\") pod \"e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.242137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.242104 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:40.247026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.246972 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:40.567139 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.566060 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr"] Nov 02 13:53:40.577678 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.577640 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:40.577802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.577740 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:40.578206 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.578177 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:40.628892 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.628859 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:40.648460 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.648428 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" event={"ID":"1baef57a-a73f-40bc-9acc-e3f13ad84a1a","Type":"ContainerStarted","Data":"557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9"} Nov 02 13:53:40.648598 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.648479 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" event={"ID":"1baef57a-a73f-40bc-9acc-e3f13ad84a1a","Type":"ContainerStarted","Data":"0cc1d9973dd2cc81a0750ac9b641231e959b1323335b5ec2947f227bdff85f33"} Nov 02 13:53:40.694471 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.694418 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podStartSLOduration=1.6944057209999999 podStartE2EDuration="1.694405721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:40.693318781 +0000 UTC m=+569.418940556" watchObservedRunningTime="2026-11-02 13:53:40.694405721 +0000 UTC m=+569.420027491" Nov 02 13:53:40.983629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.983594 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk"] Nov 02 13:53:40.987397 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:40.987376 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.005529 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.005503 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ebfc3db5-openshift-default-dockercfg-6lk45\"" Nov 02 13:53:41.016603 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.016571 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk"] Nov 02 13:53:41.072394 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072360 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7w2b\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-kube-api-access-k7w2b\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.072394 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072397 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-token\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.072602 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072420 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-socket\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.072602 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072561 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-data\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.072602 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072583 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-podinfo\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.072705 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072670 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-certs\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.072739 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072721 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-credential-socket\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.072772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072738 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-envoy\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.072805 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.072770 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb9de3e-bd88-4952-89a5-32692d236a0f-istiod-ca-cert\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.108129 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.108082 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:41.174305 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174269 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-data\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174305 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174304 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-podinfo\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174327 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-certs\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174359 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-credential-socket\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174375 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-envoy\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174418 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb9de3e-bd88-4952-89a5-32692d236a0f-istiod-ca-cert\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174452 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7w2b\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-kube-api-access-k7w2b\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174477 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-token\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174506 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-socket\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174885 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174722 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-data\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174885 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174733 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-certs\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.174885 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174805 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-credential-socket\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.175046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.174971 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-socket\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.175295 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.175276 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb9de3e-bd88-4952-89a5-32692d236a0f-istiod-ca-cert\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.176995 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.176968 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-envoy\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.177111 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.177096 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-podinfo\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.199426 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.199314 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-token\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.202606 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.202546 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7w2b\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-kube-api-access-k7w2b\") pod \"e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.247466 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.247379 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:41.252511 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.252483 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:41.271651 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.271619 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:41.300318 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.300285 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:41.618357 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.618159 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk"] Nov 02 13:53:41.620554 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:41.620524 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb9de3e_bd88_4952_89a5_32692d236a0f.slice/crio-5d4070d52ba44251641d8468ca49843acb1e8925fe2c19b01dfd2fbedd467161 WatchSource:0}: Error finding container 5d4070d52ba44251641d8468ca49843acb1e8925fe2c19b01dfd2fbedd467161: Status 404 returned error can't find the container with id 5d4070d52ba44251641d8468ca49843acb1e8925fe2c19b01dfd2fbedd467161 Nov 02 13:53:41.622778 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.622742 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:41.622867 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.622831 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:41.622927 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.622877 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:41.654167 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.654138 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" event={"ID":"0cb9de3e-bd88-4952-89a5-32692d236a0f","Type":"ContainerStarted","Data":"5d4070d52ba44251641d8468ca49843acb1e8925fe2c19b01dfd2fbedd467161"} Nov 02 13:53:41.654467 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.654440 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:41.655476 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:41.655455 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:42.104092 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:42.104059 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:42.299477 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:42.299443 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:42.475416 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:42.475388 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:42.659660 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:42.659624 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" event={"ID":"0cb9de3e-bd88-4952-89a5-32692d236a0f","Type":"ContainerStarted","Data":"9d7440f4121f6351607709f1bbd92bbfc0ac3189c069a28b19d27e09d154f06f"} Nov 02 13:53:42.762785 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:42.762681 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podStartSLOduration=2.762667106 podStartE2EDuration="2.762667106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:42.760118467 +0000 UTC m=+571.485740238" watchObservedRunningTime="2026-11-02 13:53:42.762667106 +0000 UTC m=+571.488288875" Nov 02 13:53:42.777963 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:42.777929 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:43.301405 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.301360 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:43.306569 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.306541 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:43.580771 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.580689 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:43.664213 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.664182 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:43.665397 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.665375 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:53:43.667517 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.667494 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66"] Nov 02 13:53:43.671083 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.671065 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:43.692151 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.692122 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6232c163-dockercfg-vmdrq\"" Nov 02 13:53:43.692151 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.692132 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6232c163\"" Nov 02 13:53:43.696760 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.696732 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5"] Nov 02 13:53:43.700475 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.700449 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66"] Nov 02 13:53:43.700624 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.700569 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.737380 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.737345 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5"] Nov 02 13:53:43.760462 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.760430 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:43.792593 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.792550 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 13:53:43.792779 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.792598 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 13:53:43.792779 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.792683 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 13:53:43.792779 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.792703 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 13:53:43.792779 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.792713 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6232c163-serving-cert\"" Nov 02 13:53:43.812014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.807299 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-maas-api-tls\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.812014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.807459 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42jp\" (UniqueName: \"kubernetes.io/projected/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-kube-api-access-k42jp\") pod \"payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66\" (UID: \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:43.812014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.807609 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4a0c21-3d4e-4106-a86c-c59412b989c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.812014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.811052 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66\" (UID: \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:43.816008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.812510 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-metrics-certs\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.816008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.812760 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smftn\" (UniqueName: \"kubernetes.io/projected/0c4a0c21-3d4e-4106-a86c-c59412b989c9-kube-api-access-smftn\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.915406 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.915364 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-maas-api-tls\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.915622 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.915415 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42jp\" (UniqueName: \"kubernetes.io/projected/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-kube-api-access-k42jp\") pod \"payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66\" (UID: \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:43.915622 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.915465 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4a0c21-3d4e-4106-a86c-c59412b989c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.915622 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.915509 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66\" (UID: \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:43.915622 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.915603 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-metrics-certs\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.915830 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.915650 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smftn\" (UniqueName: \"kubernetes.io/projected/0c4a0c21-3d4e-4106-a86c-c59412b989c9-kube-api-access-smftn\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.916315 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.916285 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4a0c21-3d4e-4106-a86c-c59412b989c9-openshift-service-ca\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.916433 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.916412 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66\" (UID: \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:43.918001 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.917958 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-metrics-certs\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.918091 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.918009 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-maas-api-tls\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.925487 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.925460 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:43.938641 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.938614 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42jp\" (UniqueName: \"kubernetes.io/projected/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-kube-api-access-k42jp\") pod \"payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66\" (UID: \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:43.938742 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.938708 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smftn\" (UniqueName: \"kubernetes.io/projected/0c4a0c21-3d4e-4106-a86c-c59412b989c9-kube-api-access-smftn\") pod \"maas-api-e2e-migrate-6232c163-646b88d88d-lblx5\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:43.987302 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:43.985532 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:44.014502 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.014465 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:44.089440 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.085894 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:44.247033 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.246921 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:44.306023 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.305977 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66"] Nov 02 13:53:44.310923 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:44.310900 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9ae06c_3784_4b4d_93e6_dc71dadf11bb.slice/crio-729ee3acc46714aab2d73667ae5199e951cea3e8200738d6fdf854140d450810 WatchSource:0}: Error finding container 729ee3acc46714aab2d73667ae5199e951cea3e8200738d6fdf854140d450810: Status 404 returned error can't find the container with id 729ee3acc46714aab2d73667ae5199e951cea3e8200738d6fdf854140d450810 Nov 02 13:53:44.339282 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.339255 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5"] Nov 02 13:53:44.342318 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:44.342291 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4a0c21_3d4e_4106_a86c_c59412b989c9.slice/crio-233281493aae6191a6e5f4dd58c7c3fa4cf4de2704e7c1c3b4f7539c40a29d1d WatchSource:0}: Error finding container 233281493aae6191a6e5f4dd58c7c3fa4cf4de2704e7c1c3b4f7539c40a29d1d: Status 404 returned error can't find the container with id 233281493aae6191a6e5f4dd58c7c3fa4cf4de2704e7c1c3b4f7539c40a29d1d Nov 02 13:53:44.670457 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.670411 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" event={"ID":"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb","Type":"ContainerStarted","Data":"caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756"} Nov 02 13:53:44.670626 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.670477 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:44.670626 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.670502 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" event={"ID":"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb","Type":"ContainerStarted","Data":"729ee3acc46714aab2d73667ae5199e951cea3e8200738d6fdf854140d450810"} Nov 02 13:53:44.671869 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.671837 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" event={"ID":"0c4a0c21-3d4e-4106-a86c-c59412b989c9","Type":"ContainerStarted","Data":"233281493aae6191a6e5f4dd58c7c3fa4cf4de2704e7c1c3b4f7539c40a29d1d"} Nov 02 13:53:44.711187 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:44.711134 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" podStartSLOduration=1.711120351 podStartE2EDuration="1.711120351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:44.710161453 +0000 UTC m=+573.435783234" watchObservedRunningTime="2026-11-02 13:53:44.711120351 +0000 UTC m=+573.436742112" Nov 02 13:53:45.349561 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:45.349529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:45.550019 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:45.549919 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:45.677907 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:45.677870 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" event={"ID":"0c4a0c21-3d4e-4106-a86c-c59412b989c9","Type":"ContainerStarted","Data":"c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921"} Nov 02 13:53:45.678134 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:45.678111 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:45.718427 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:45.718394 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:45.726086 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:45.726038 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" podStartSLOduration=2.3128925320000002 podStartE2EDuration="2.726025779s" totalImagesPullingTime="413.133247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:45.725230239 +0000 UTC m=+574.450852008" watchObservedRunningTime="2026-11-02 13:53:45.726025779 +0000 UTC m=+574.451647549" Nov 02 13:53:45.884754 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:45.884708 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:46.456026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.453225 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:46.490175 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.490138 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw"] Nov 02 13:53:46.493789 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.493769 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:46.520727 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.520694 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c57ba6aa\"" Nov 02 13:53:46.520883 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.520731 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c57ba6aa-dockercfg-2cwdj\"" Nov 02 13:53:46.524198 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.524172 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw"] Nov 02 13:53:46.581133 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.581096 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z"] Nov 02 13:53:46.584848 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.584821 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:46.609489 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.609396 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z"] Nov 02 13:53:46.636484 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.636449 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:46.640404 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.640370 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:53:46.663029 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.660534 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cccbb2-d292-47bf-be37-4b3c244054a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw\" (UID: \"78cccbb2-d292-47bf-be37-4b3c244054a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:46.663029 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.660590 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsp5t\" (UniqueName: \"kubernetes.io/projected/78cccbb2-d292-47bf-be37-4b3c244054a8-kube-api-access-qsp5t\") pod \"payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw\" (UID: \"78cccbb2-d292-47bf-be37-4b3c244054a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:46.663029 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.660663 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4jd\" (UniqueName: \"kubernetes.io/projected/752cca65-73af-43de-9959-36c6bcf2b750-kube-api-access-dm4jd\") pod \"payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z\" (UID: \"752cca65-73af-43de-9959-36c6bcf2b750\") " pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:46.663029 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.661866 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cca65-73af-43de-9959-36c6bcf2b750-plugins-config-volume\") pod \"payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z\" (UID: \"752cca65-73af-43de-9959-36c6bcf2b750\") " pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:46.762973 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.762893 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cca65-73af-43de-9959-36c6bcf2b750-plugins-config-volume\") pod \"payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z\" (UID: \"752cca65-73af-43de-9959-36c6bcf2b750\") " pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:46.762973 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.762955 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cccbb2-d292-47bf-be37-4b3c244054a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw\" (UID: \"78cccbb2-d292-47bf-be37-4b3c244054a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:46.763192 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.762975 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsp5t\" (UniqueName: \"kubernetes.io/projected/78cccbb2-d292-47bf-be37-4b3c244054a8-kube-api-access-qsp5t\") pod \"payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw\" (UID: \"78cccbb2-d292-47bf-be37-4b3c244054a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:46.763256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.763191 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4jd\" (UniqueName: \"kubernetes.io/projected/752cca65-73af-43de-9959-36c6bcf2b750-kube-api-access-dm4jd\") pod \"payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z\" (UID: \"752cca65-73af-43de-9959-36c6bcf2b750\") " pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:46.763809 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.763783 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cccbb2-d292-47bf-be37-4b3c244054a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw\" (UID: \"78cccbb2-d292-47bf-be37-4b3c244054a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:46.763899 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.763815 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cca65-73af-43de-9959-36c6bcf2b750-plugins-config-volume\") pod \"payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z\" (UID: \"752cca65-73af-43de-9959-36c6bcf2b750\") " pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:46.786752 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.786712 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsp5t\" (UniqueName: \"kubernetes.io/projected/78cccbb2-d292-47bf-be37-4b3c244054a8-kube-api-access-qsp5t\") pod \"payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw\" (UID: \"78cccbb2-d292-47bf-be37-4b3c244054a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:46.786883 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.786730 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4jd\" (UniqueName: \"kubernetes.io/projected/752cca65-73af-43de-9959-36c6bcf2b750-kube-api-access-dm4jd\") pod \"payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z\" (UID: \"752cca65-73af-43de-9959-36c6bcf2b750\") " pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:46.807542 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.807506 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:46.899338 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:46.899303 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:47.117835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.117805 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw"] Nov 02 13:53:47.120258 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:47.120230 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cccbb2_d292_47bf_be37_4b3c244054a8.slice/crio-ce5a1b07cebc2c982a167954702a077f2f2a10a2b701129bafbef8e5d31a4697 WatchSource:0}: Error finding container ce5a1b07cebc2c982a167954702a077f2f2a10a2b701129bafbef8e5d31a4697: Status 404 returned error can't find the container with id ce5a1b07cebc2c982a167954702a077f2f2a10a2b701129bafbef8e5d31a4697 Nov 02 13:53:47.222202 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.222172 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z"] Nov 02 13:53:47.224882 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:47.224852 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752cca65_73af_43de_9959_36c6bcf2b750.slice/crio-1f4197540154154e59b6afb8d28ff34783ff14c2f509d5928a082ddee8398db3 WatchSource:0}: Error finding container 1f4197540154154e59b6afb8d28ff34783ff14c2f509d5928a082ddee8398db3: Status 404 returned error can't find the container with id 1f4197540154154e59b6afb8d28ff34783ff14c2f509d5928a082ddee8398db3 Nov 02 13:53:47.688407 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.688362 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" event={"ID":"78cccbb2-d292-47bf-be37-4b3c244054a8","Type":"ContainerStarted","Data":"776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578"} Nov 02 13:53:47.688849 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.688423 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" event={"ID":"78cccbb2-d292-47bf-be37-4b3c244054a8","Type":"ContainerStarted","Data":"ce5a1b07cebc2c982a167954702a077f2f2a10a2b701129bafbef8e5d31a4697"} Nov 02 13:53:47.688849 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.688525 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:47.689871 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.689851 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" event={"ID":"752cca65-73af-43de-9959-36c6bcf2b750","Type":"ContainerStarted","Data":"fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c"} Nov 02 13:53:47.689955 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.689887 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" event={"ID":"752cca65-73af-43de-9959-36c6bcf2b750","Type":"ContainerStarted","Data":"1f4197540154154e59b6afb8d28ff34783ff14c2f509d5928a082ddee8398db3"} Nov 02 13:53:47.689955 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.689907 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:47.758871 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.758799 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" podStartSLOduration=1.7587787179999999 podStartE2EDuration="1.758778718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:47.75651523 +0000 UTC m=+576.482137002" watchObservedRunningTime="2026-11-02 13:53:47.758778718 +0000 UTC m=+576.484400492" Nov 02 13:53:47.798791 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.798739 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" podStartSLOduration=1.798726978 podStartE2EDuration="1.798726978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:47.798093094 +0000 UTC m=+576.523714877" watchObservedRunningTime="2026-11-02 13:53:47.798726978 +0000 UTC m=+576.524348747" Nov 02 13:53:47.825276 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.825240 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:47.995499 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:47.995406 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:48.166643 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:48.166606 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:48.356059 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:48.354734 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:48.796481 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:48.796448 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:48.963395 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:48.963358 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:49.115823 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.115740 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v"] Nov 02 13:53:49.119387 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.119371 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:49.138389 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.138358 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ebfc3db5\"" Nov 02 13:53:49.147559 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.147527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:49.153689 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.153658 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v"] Nov 02 13:53:49.154067 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.154045 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ebfc3db5-dockercfg-mmz4c\"" Nov 02 13:53:49.202181 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.202146 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz"] Nov 02 13:53:49.209937 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.209909 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:49.264388 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.264340 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz"] Nov 02 13:53:49.291518 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.291484 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v\" (UID: \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:49.291703 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.291681 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bns\" (UniqueName: \"kubernetes.io/projected/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-kube-api-access-t2bns\") pod \"payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v\" (UID: \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:49.392665 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.392633 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lkt\" (UniqueName: \"kubernetes.io/projected/33981cdf-9fc8-45a1-af5d-cc327b216099-kube-api-access-57lkt\") pod \"payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz\" (UID: \"33981cdf-9fc8-45a1-af5d-cc327b216099\") " pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:49.392847 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.392706 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33981cdf-9fc8-45a1-af5d-cc327b216099-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz\" (UID: \"33981cdf-9fc8-45a1-af5d-cc327b216099\") " pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:49.392847 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.392797 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bns\" (UniqueName: \"kubernetes.io/projected/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-kube-api-access-t2bns\") pod \"payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v\" (UID: \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:49.392847 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.392840 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v\" (UID: \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:49.393869 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.393849 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v\" (UID: \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:49.416522 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.416491 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bns\" (UniqueName: \"kubernetes.io/projected/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-kube-api-access-t2bns\") pod \"payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v\" (UID: \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:49.461543 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.461508 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:49.493529 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.493487 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lkt\" (UniqueName: \"kubernetes.io/projected/33981cdf-9fc8-45a1-af5d-cc327b216099-kube-api-access-57lkt\") pod \"payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz\" (UID: \"33981cdf-9fc8-45a1-af5d-cc327b216099\") " pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:49.493732 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.493560 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33981cdf-9fc8-45a1-af5d-cc327b216099-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz\" (UID: \"33981cdf-9fc8-45a1-af5d-cc327b216099\") " pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:49.494275 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.494250 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33981cdf-9fc8-45a1-af5d-cc327b216099-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz\" (UID: \"33981cdf-9fc8-45a1-af5d-cc327b216099\") " pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:49.518835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.518810 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lkt\" (UniqueName: \"kubernetes.io/projected/33981cdf-9fc8-45a1-af5d-cc327b216099-kube-api-access-57lkt\") pod \"payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz\" (UID: \"33981cdf-9fc8-45a1-af5d-cc327b216099\") " pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:49.580296 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.579794 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:49.805590 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.805559 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v"] Nov 02 13:53:49.807926 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:49.807899 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe9095b_b3c3_45b4_aa20_7b24fdf6af57.slice/crio-6e86f3f670701ddcb717233c0ccefe565eab09b59b51c62890ef40175251523b WatchSource:0}: Error finding container 6e86f3f670701ddcb717233c0ccefe565eab09b59b51c62890ef40175251523b: Status 404 returned error can't find the container with id 6e86f3f670701ddcb717233c0ccefe565eab09b59b51c62890ef40175251523b Nov 02 13:53:49.954071 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:49.948781 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz"] Nov 02 13:53:49.961962 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:49.961932 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33981cdf_9fc8_45a1_af5d_cc327b216099.slice/crio-093cf4e02dda3b92f7b6d24304b2ca422326f9dae00ebf53224e4d3e239c0b03 WatchSource:0}: Error finding container 093cf4e02dda3b92f7b6d24304b2ca422326f9dae00ebf53224e4d3e239c0b03: Status 404 returned error can't find the container with id 093cf4e02dda3b92f7b6d24304b2ca422326f9dae00ebf53224e4d3e239c0b03 Nov 02 13:53:50.026970 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.024843 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:50.207716 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.207645 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:50.397267 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.397232 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:50.708548 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.708514 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" event={"ID":"33981cdf-9fc8-45a1-af5d-cc327b216099","Type":"ContainerStarted","Data":"642904437cb0b768ef9c06dee1d7a2affed508d1b0f2000bd4f123937ff6153e"} Nov 02 13:53:50.708751 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.708568 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" event={"ID":"33981cdf-9fc8-45a1-af5d-cc327b216099","Type":"ContainerStarted","Data":"093cf4e02dda3b92f7b6d24304b2ca422326f9dae00ebf53224e4d3e239c0b03"} Nov 02 13:53:50.708751 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.708601 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:53:50.710336 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.710308 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" event={"ID":"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57","Type":"ContainerStarted","Data":"641331fb9f7dcedb2b63c68f47f03185f5eec7a4838b41d4bff77c0254c54dcc"} Nov 02 13:53:50.710444 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.710359 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" event={"ID":"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57","Type":"ContainerStarted","Data":"6e86f3f670701ddcb717233c0ccefe565eab09b59b51c62890ef40175251523b"} Nov 02 13:53:50.710444 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.710395 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:53:50.754014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.753580 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" podStartSLOduration=1.753561136 podStartE2EDuration="1.753561136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:50.751466421 +0000 UTC m=+579.477088198" watchObservedRunningTime="2026-11-02 13:53:50.753561136 +0000 UTC m=+579.479182907" Nov 02 13:53:50.792149 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:50.792090 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" podStartSLOduration=1.7920724799999999 podStartE2EDuration="1.79207248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:50.791717719 +0000 UTC m=+579.517339488" watchObservedRunningTime="2026-11-02 13:53:50.79207248 +0000 UTC m=+579.517694252" Nov 02 13:53:51.257118 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:51.257050 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:51.471371 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:51.471337 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:51.691933 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:51.691900 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:53:52.220268 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:52.220230 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:53.872263 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:53.872222 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:53.952767 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:53.952727 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz"] Nov 02 13:53:53.957954 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:53.957923 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:53.987488 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:53.987400 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e7818088\"" Nov 02 13:53:53.987740 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:53.987507 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e7818088-dockercfg-xbtpm\"" Nov 02 13:53:53.991510 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:53.991476 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz"] Nov 02 13:53:53.997220 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:53.997189 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7639b-4680-4d6f-b679-7a41f28f9277-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz\" (UID: \"b3b7639b-4680-4d6f-b679-7a41f28f9277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:53.997630 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:53.997607 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866bh\" (UniqueName: \"kubernetes.io/projected/b3b7639b-4680-4d6f-b679-7a41f28f9277-kube-api-access-866bh\") pod \"payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz\" (UID: \"b3b7639b-4680-4d6f-b679-7a41f28f9277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:54.100477 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.098797 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866bh\" (UniqueName: \"kubernetes.io/projected/b3b7639b-4680-4d6f-b679-7a41f28f9277-kube-api-access-866bh\") pod \"payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz\" (UID: \"b3b7639b-4680-4d6f-b679-7a41f28f9277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:54.100477 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.098861 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7639b-4680-4d6f-b679-7a41f28f9277-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz\" (UID: \"b3b7639b-4680-4d6f-b679-7a41f28f9277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:54.100477 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.099794 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7639b-4680-4d6f-b679-7a41f28f9277-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz\" (UID: \"b3b7639b-4680-4d6f-b679-7a41f28f9277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:54.117307 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.117262 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:54.128824 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.128740 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866bh\" (UniqueName: \"kubernetes.io/projected/b3b7639b-4680-4d6f-b679-7a41f28f9277-kube-api-access-866bh\") pod \"payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz\" (UID: \"b3b7639b-4680-4d6f-b679-7a41f28f9277\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:54.273227 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.273186 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:54.470050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.469998 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:54.628282 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.628246 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz"] Nov 02 13:53:54.628469 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:53:54.628444 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b7639b_4680_4d6f_b679_7a41f28f9277.slice/crio-e5fe260dc56781316e53857f4d279dd5c114b697f6fc59333b7e387e28af25aa WatchSource:0}: Error finding container e5fe260dc56781316e53857f4d279dd5c114b697f6fc59333b7e387e28af25aa: Status 404 returned error can't find the container with id e5fe260dc56781316e53857f4d279dd5c114b697f6fc59333b7e387e28af25aa Nov 02 13:53:54.756055 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.755220 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" event={"ID":"b3b7639b-4680-4d6f-b679-7a41f28f9277","Type":"ContainerStarted","Data":"051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356"} Nov 02 13:53:54.756055 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.755290 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" event={"ID":"b3b7639b-4680-4d6f-b679-7a41f28f9277","Type":"ContainerStarted","Data":"e5fe260dc56781316e53857f4d279dd5c114b697f6fc59333b7e387e28af25aa"} Nov 02 13:53:54.756345 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.756145 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:53:54.820903 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:54.820833 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" podStartSLOduration=1.8208149329999999 podStartE2EDuration="1.820814933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:54.819170204 +0000 UTC m=+583.544791974" watchObservedRunningTime="2026-11-02 13:53:54.820814933 +0000 UTC m=+583.546436705" Nov 02 13:53:55.684718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:55.683861 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:53:56.607759 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:56.607713 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:56.877544 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:56.877175 5591 patch_prober.go:29] interesting pod/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:50364->10.133.0.33:15021: read: connection reset by peer" start-of-body= Nov 02 13:53:56.877544 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:56.877250 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:50364->10.133.0.33:15021: read: connection reset by peer" Nov 02 13:53:57.054865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:57.054020 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:53:57.782596 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:57.782562 5591 generic.go:356] "Generic (PLEG): container finished" podID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerID="557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9" exitCode=0 Nov 02 13:53:57.782758 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:57.782637 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" event={"ID":"1baef57a-a73f-40bc-9acc-e3f13ad84a1a","Type":"ContainerDied","Data":"557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9"} Nov 02 13:53:57.783062 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:57.783039 5591 scope.go:122] "RemoveContainer" containerID="557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9" Nov 02 13:53:57.784321 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:57.784294 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:57.784387 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:57.784364 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:57.784425 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:57.784396 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:53:58.697488 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:58.697454 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:58.698244 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:58.698221 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:53:58.698376 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:58.698304 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:53:58.791936 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:58.791901 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" event={"ID":"1baef57a-a73f-40bc-9acc-e3f13ad84a1a","Type":"ContainerStarted","Data":"966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74"} Nov 02 13:53:59.259032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:59.248392 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:53:59.259032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:59.248519 5591 patch_prober.go:29] interesting pod/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 13:53:59.259032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:59.248570 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 13:53:59.705155 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:59.705114 5591 patch_prober.go:29] interesting pod/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:53:59.705578 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:53:59.705213 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:54:00.251488 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.251435 5591 patch_prober.go:29] interesting pod/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 13:54:00.251782 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.251746 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 13:54:00.662004 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.660913 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7"] Nov 02 13:54:00.664318 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.664292 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.680181 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.680142 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e7818088-serving-cert\"" Nov 02 13:54:00.690499 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.690467 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7"] Nov 02 13:54:00.795388 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.795354 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:00.814410 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.814373 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-metrics-certs\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.814576 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.814442 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.814576 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.814539 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf283da-d37b-4518-b1da-dbe242c510e7-openshift-service-ca\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.814663 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.814642 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgn84\" (UniqueName: \"kubernetes.io/projected/2bf283da-d37b-4518-b1da-dbe242c510e7-kube-api-access-kgn84\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.916104 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.916007 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgn84\" (UniqueName: \"kubernetes.io/projected/2bf283da-d37b-4518-b1da-dbe242c510e7-kube-api-access-kgn84\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.916104 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.916077 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-metrics-certs\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.916354 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.916226 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.916354 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.916272 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf283da-d37b-4518-b1da-dbe242c510e7-openshift-service-ca\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.916466 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:00.916381 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-serving-cert: secret "maas-api-e2e-derive-e7818088-serving-cert" not found Nov 02 13:54:00.916518 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:00.916469 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls podName:2bf283da-d37b-4518-b1da-dbe242c510e7 nodeName:}" failed. No retries permitted until 2026-11-02 13:54:01.4164481 +0000 UTC m=+590.142069857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls") pod "maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" (UID: "2bf283da-d37b-4518-b1da-dbe242c510e7") : secret "maas-api-e2e-derive-e7818088-serving-cert" not found Nov 02 13:54:00.916886 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.916867 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf283da-d37b-4518-b1da-dbe242c510e7-openshift-service-ca\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.918601 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.918578 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-metrics-certs\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.957184 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.957147 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgn84\" (UniqueName: \"kubernetes.io/projected/2bf283da-d37b-4518-b1da-dbe242c510e7-kube-api-access-kgn84\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:00.985034 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:00.984973 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:01.085251 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:01.085218 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:54:01.166663 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.166564 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:01.250713 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.250672 5591 patch_prober.go:29] interesting pod/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 13:54:01.251036 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.250975 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 13:54:01.422597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.422501 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:01.426268 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.426238 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls\") pod \"maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:01.576249 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.576213 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:01.718148 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.718065 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:54:01.718148 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.718129 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:54:01.813490 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.813015 5591 generic.go:356] "Generic (PLEG): container finished" podID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerID="9d7440f4121f6351607709f1bbd92bbfc0ac3189c069a28b19d27e09d154f06f" exitCode=0 Nov 02 13:54:01.813490 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.813125 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" event={"ID":"0cb9de3e-bd88-4952-89a5-32692d236a0f","Type":"ContainerDied","Data":"9d7440f4121f6351607709f1bbd92bbfc0ac3189c069a28b19d27e09d154f06f"} Nov 02 13:54:01.814083 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.813512 5591 scope.go:122] "RemoveContainer" containerID="9d7440f4121f6351607709f1bbd92bbfc0ac3189c069a28b19d27e09d154f06f" Nov 02 13:54:01.814965 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.814888 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:01.815217 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.815190 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:01.815387 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.815364 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:01.893396 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:01.893364 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7"] Nov 02 13:54:02.267005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:02.262511 5591 patch_prober.go:29] interesting pod/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 13:54:02.267005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:02.262573 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 13:54:02.693656 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:02.693588 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:02.820464 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:02.820423 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" event={"ID":"0cb9de3e-bd88-4952-89a5-32692d236a0f","Type":"ContainerStarted","Data":"39bd8acae5b8ae2177aad45df4e0f097031733c12c98c90ce8c7b4e62895bfc0"} Nov 02 13:54:02.822116 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:02.822090 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" event={"ID":"2bf283da-d37b-4518-b1da-dbe242c510e7","Type":"ContainerStarted","Data":"655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531"} Nov 02 13:54:02.822299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:02.822149 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" event={"ID":"2bf283da-d37b-4518-b1da-dbe242c510e7","Type":"ContainerStarted","Data":"1454788eee21c5053326fd9b9acda2983a69be9e6d79919a9fe5d9a9a9c40913"} Nov 02 13:54:02.822299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:02.822266 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:02.953067 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:02.932585 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" podStartSLOduration=2.602626705 podStartE2EDuration="2.932565203s" totalImagesPullingTime="329.938498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:02.930159731 +0000 UTC m=+591.655781501" watchObservedRunningTime="2026-11-02 13:54:02.932565203 +0000 UTC m=+591.658186973" Nov 02 13:54:03.180069 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:03.180033 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:03.251089 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:03.250974 5591 patch_prober.go:29] interesting pod/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 13:54:03.251351 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:03.251324 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 13:54:03.300858 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:03.300825 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:54:03.301426 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:03.301401 5591 patch_prober.go:29] interesting pod/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 13:54:03.301490 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:03.301450 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 13:54:03.355955 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:03.355917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:04.252007 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:04.251543 5591 patch_prober.go:29] interesting pod/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 13:54:04.252007 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:04.251622 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 13:54:04.301162 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:04.301121 5591 patch_prober.go:29] interesting pod/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 13:54:04.301345 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:04.301189 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 13:54:04.423606 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:04.423570 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:54:04.494818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:04.494787 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:05.260189 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:05.260152 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:54:05.260775 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:05.260756 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:54:05.261896 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:05.261873 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:54:05.301309 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:05.301276 5591 patch_prober.go:29] interesting pod/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 13:54:05.301528 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:05.301349 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 13:54:06.300809 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:06.300768 5591 patch_prober.go:29] interesting pod/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 13:54:06.301314 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:06.300828 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 13:54:06.391679 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:06.391640 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:06.779612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:06.779583 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:54:07.301643 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:07.301605 5591 patch_prober.go:29] interesting pod/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 13:54:07.302103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:07.301672 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 13:54:08.220639 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:08.220599 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:08.306104 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:08.306067 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:54:08.306574 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:08.306385 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:54:08.307065 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:08.307042 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:54:08.834087 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:08.834058 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:10.101379 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:10.101343 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:11.523637 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:11.523600 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:54:11.524320 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:11.524299 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:54:11.998230 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:11.998194 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:13.699814 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:13.699784 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:15.782101 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:15.782064 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz"] Nov 02 13:54:15.801173 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:15.801144 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v"] Nov 02 13:54:15.826185 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:15.826153 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:16.100755 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:16.100674 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:17.848465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:17.848394 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:17.919148 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:17.919116 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66"] Nov 02 13:54:18.097689 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:18.097652 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:18.262206 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:18.262176 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:18.901023 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:18.900992 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:19.084325 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:19.084291 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:19.706900 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:19.706864 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:20.414332 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.414301 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:20.689595 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.689512 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:20.783531 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.783473 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" podUID="33981cdf-9fc8-45a1-af5d-cc327b216099" containerName="payload-processing" containerID="cri-o://642904437cb0b768ef9c06dee1d7a2affed508d1b0f2000bd4f123937ff6153e" gracePeriod=25 Nov 02 13:54:20.802060 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.802029 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" podUID="bfe9095b-b3c3-45b4-aa20-7b24fdf6af57" containerName="payload-pre-processing" containerID="cri-o://641331fb9f7dcedb2b63c68f47f03185f5eec7a4838b41d4bff77c0254c54dcc" gracePeriod=25 Nov 02 13:54:20.946933 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.946856 5591 generic.go:356] "Generic (PLEG): container finished" podID="bfe9095b-b3c3-45b4-aa20-7b24fdf6af57" containerID="641331fb9f7dcedb2b63c68f47f03185f5eec7a4838b41d4bff77c0254c54dcc" exitCode=0 Nov 02 13:54:20.947104 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.946929 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" event={"ID":"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57","Type":"ContainerDied","Data":"641331fb9f7dcedb2b63c68f47f03185f5eec7a4838b41d4bff77c0254c54dcc"} Nov 02 13:54:20.948373 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.948354 5591 generic.go:356] "Generic (PLEG): container finished" podID="33981cdf-9fc8-45a1-af5d-cc327b216099" containerID="642904437cb0b768ef9c06dee1d7a2affed508d1b0f2000bd4f123937ff6153e" exitCode=0 Nov 02 13:54:20.948433 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.948397 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" event={"ID":"33981cdf-9fc8-45a1-af5d-cc327b216099","Type":"ContainerDied","Data":"642904437cb0b768ef9c06dee1d7a2affed508d1b0f2000bd4f123937ff6153e"} Nov 02 13:54:20.986507 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:20.986467 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:21.048685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.048657 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:54:21.069394 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.069372 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:54:21.176576 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.176538 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33981cdf-9fc8-45a1-af5d-cc327b216099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33981cdf-9fc8-45a1-af5d-cc327b216099-plugins-config-volume\") pod \"33981cdf-9fc8-45a1-af5d-cc327b216099\" (UID: \"33981cdf-9fc8-45a1-af5d-cc327b216099\") " Nov 02 13:54:21.176749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.176595 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-kube-api-access-t2bns\" (UniqueName: \"kubernetes.io/projected/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-kube-api-access-t2bns\") pod \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\" (UID: \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\") " Nov 02 13:54:21.176749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.176702 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33981cdf-9fc8-45a1-af5d-cc327b216099-kube-api-access-57lkt\" (UniqueName: \"kubernetes.io/projected/33981cdf-9fc8-45a1-af5d-cc327b216099-kube-api-access-57lkt\") pod \"33981cdf-9fc8-45a1-af5d-cc327b216099\" (UID: \"33981cdf-9fc8-45a1-af5d-cc327b216099\") " Nov 02 13:54:21.176836 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.176815 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-plugins-config-volume\") pod \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\" (UID: \"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57\") " Nov 02 13:54:21.176930 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.176909 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33981cdf-9fc8-45a1-af5d-cc327b216099-plugins-config-volume" pod "33981cdf-9fc8-45a1-af5d-cc327b216099" (UID: "33981cdf-9fc8-45a1-af5d-cc327b216099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:21.177174 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.177155 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33981cdf-9fc8-45a1-af5d-cc327b216099-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:21.177270 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.177229 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-plugins-config-volume" pod "bfe9095b-b3c3-45b4-aa20-7b24fdf6af57" (UID: "bfe9095b-b3c3-45b4-aa20-7b24fdf6af57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:21.178712 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.178686 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-kube-api-access-t2bns" pod "bfe9095b-b3c3-45b4-aa20-7b24fdf6af57" (UID: "bfe9095b-b3c3-45b4-aa20-7b24fdf6af57"). InnerVolumeSpecName "kube-api-access-t2bns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:21.178817 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.178728 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33981cdf-9fc8-45a1-af5d-cc327b216099-kube-api-access-57lkt" pod "33981cdf-9fc8-45a1-af5d-cc327b216099" (UID: "33981cdf-9fc8-45a1-af5d-cc327b216099"). InnerVolumeSpecName "kube-api-access-57lkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:21.277854 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.277772 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2bns\" (UniqueName: \"kubernetes.io/projected/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-kube-api-access-t2bns\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:21.277854 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.277801 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57lkt\" (UniqueName: \"kubernetes.io/projected/33981cdf-9fc8-45a1-af5d-cc327b216099-kube-api-access-57lkt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:21.277854 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.277811 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:21.432627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.430826 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:21.954121 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.954093 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" Nov 02 13:54:21.954288 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.954121 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v" event={"ID":"bfe9095b-b3c3-45b4-aa20-7b24fdf6af57","Type":"ContainerDied","Data":"6e86f3f670701ddcb717233c0ccefe565eab09b59b51c62890ef40175251523b"} Nov 02 13:54:21.954288 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.954177 5591 scope.go:122] "RemoveContainer" containerID="641331fb9f7dcedb2b63c68f47f03185f5eec7a4838b41d4bff77c0254c54dcc" Nov 02 13:54:21.955808 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.955783 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" event={"ID":"33981cdf-9fc8-45a1-af5d-cc327b216099","Type":"ContainerDied","Data":"093cf4e02dda3b92f7b6d24304b2ca422326f9dae00ebf53224e4d3e239c0b03"} Nov 02 13:54:21.955907 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.955846 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz" Nov 02 13:54:21.962816 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:21.962796 5591 scope.go:122] "RemoveContainer" containerID="642904437cb0b768ef9c06dee1d7a2affed508d1b0f2000bd4f123937ff6153e" Nov 02 13:54:22.008535 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.008506 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v"] Nov 02 13:54:22.021145 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.021115 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ebfc3db5-5b98df7df5-snd6v"] Nov 02 13:54:22.045751 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.045676 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:22.065277 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.065238 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz"] Nov 02 13:54:22.080278 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.080246 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebfc3db5-65856d8c99-jhtqz"] Nov 02 13:54:22.284331 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.284297 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:22.486216 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.486184 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:22.850883 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.850800 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk"] Nov 02 13:54:22.851049 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.851011 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" containerID="cri-o://39bd8acae5b8ae2177aad45df4e0f097031733c12c98c90ce8c7b4e62895bfc0" gracePeriod=30 Nov 02 13:54:22.919771 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.919720 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" podUID="2d9ae06c-3784-4b4d-93e6-dc71dadf11bb" containerName="payload-pre-processing" containerID="cri-o://caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756" gracePeriod=25 Nov 02 13:54:22.959865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:22.959828 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:23.177688 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.177661 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:54:23.308002 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.307960 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-kube-api-access-k42jp\" (UniqueName: \"kubernetes.io/projected/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-kube-api-access-k42jp\") pod \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\" (UID: \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\") " Nov 02 13:54:23.308198 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.308062 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-plugins-config-volume\") pod \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\" (UID: \"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb\") " Nov 02 13:54:23.308465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.308440 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-plugins-config-volume" pod "2d9ae06c-3784-4b4d-93e6-dc71dadf11bb" (UID: "2d9ae06c-3784-4b4d-93e6-dc71dadf11bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:23.308755 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.308736 5591 patch_prober.go:29] interesting pod/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 02 13:54:23.308823 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.308786 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 13:54:23.310158 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.310135 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-kube-api-access-k42jp" pod "2d9ae06c-3784-4b4d-93e6-dc71dadf11bb" (UID: "2d9ae06c-3784-4b4d-93e6-dc71dadf11bb"). InnerVolumeSpecName "kube-api-access-k42jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:23.408944 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.408855 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42jp\" (UniqueName: \"kubernetes.io/projected/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-kube-api-access-k42jp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:23.408944 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.408882 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:23.429308 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.429268 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33981cdf-9fc8-45a1-af5d-cc327b216099" path="/var/lib/kubelet/pods/33981cdf-9fc8-45a1-af5d-cc327b216099/volumes" Nov 02 13:54:23.429908 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.429744 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe9095b-b3c3-45b4-aa20-7b24fdf6af57" path="/var/lib/kubelet/pods/bfe9095b-b3c3-45b4-aa20-7b24fdf6af57/volumes" Nov 02 13:54:23.623018 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.622956 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z"] Nov 02 13:54:23.661884 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.661785 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw"] Nov 02 13:54:23.719502 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.719468 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:23.911373 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.911343 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:23.977953 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.977876 5591 generic.go:356] "Generic (PLEG): container finished" podID="2d9ae06c-3784-4b4d-93e6-dc71dadf11bb" containerID="caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756" exitCode=0 Nov 02 13:54:23.977953 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.977920 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" Nov 02 13:54:23.977953 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.977943 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" event={"ID":"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb","Type":"ContainerDied","Data":"caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756"} Nov 02 13:54:23.978191 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.978010 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66" event={"ID":"2d9ae06c-3784-4b4d-93e6-dc71dadf11bb","Type":"ContainerDied","Data":"729ee3acc46714aab2d73667ae5199e951cea3e8200738d6fdf854140d450810"} Nov 02 13:54:23.978191 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.978041 5591 scope.go:122] "RemoveContainer" containerID="caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756" Nov 02 13:54:23.986330 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.986309 5591 scope.go:122] "RemoveContainer" containerID="caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756" Nov 02 13:54:23.986600 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:23.986577 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756\": container with ID starting with caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756 not found: ID does not exist" containerID="caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756" Nov 02 13:54:23.986674 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:23.986613 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756"} err="failed to get container status \"caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756\": rpc error: code = NotFound desc = could not find container \"caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756\": container with ID starting with caf16612143be8a7bb6a864e4e8d06a36100f4583147204c512d97f4a6ce5756 not found: ID does not exist" Nov 02 13:54:24.039602 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.039570 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66"] Nov 02 13:54:24.050816 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.050787 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6232c163-7cfbcbd4f9-zcl66"] Nov 02 13:54:24.113443 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.113414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:24.280948 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.280856 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:24.353952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.353919 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5"] Nov 02 13:54:24.354566 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.354531 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" podUID="0c4a0c21-3d4e-4106-a86c-c59412b989c9" containerName="maas-api" containerID="cri-o://c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921" gracePeriod=30 Nov 02 13:54:24.481516 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.481486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:24.655963 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.655930 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:54:24.722021 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.721969 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4a0c21-3d4e-4106-a86c-c59412b989c9-kube-api-access-smftn\" (UniqueName: \"kubernetes.io/projected/0c4a0c21-3d4e-4106-a86c-c59412b989c9-kube-api-access-smftn\") pod \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " Nov 02 13:54:24.722224 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.722056 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-metrics-certs\") pod \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " Nov 02 13:54:24.722224 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.722117 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4a0c21-3d4e-4106-a86c-c59412b989c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4a0c21-3d4e-4106-a86c-c59412b989c9-openshift-service-ca\") pod \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " Nov 02 13:54:24.722224 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.722169 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-maas-api-tls\") pod \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\" (UID: \"0c4a0c21-3d4e-4106-a86c-c59412b989c9\") " Nov 02 13:54:24.722521 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.722491 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4a0c21-3d4e-4106-a86c-c59412b989c9-openshift-service-ca" pod "0c4a0c21-3d4e-4106-a86c-c59412b989c9" (UID: "0c4a0c21-3d4e-4106-a86c-c59412b989c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:24.724094 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.724068 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4a0c21-3d4e-4106-a86c-c59412b989c9-kube-api-access-smftn" pod "0c4a0c21-3d4e-4106-a86c-c59412b989c9" (UID: "0c4a0c21-3d4e-4106-a86c-c59412b989c9"). InnerVolumeSpecName "kube-api-access-smftn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:24.724202 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.724177 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-metrics-certs" pod "0c4a0c21-3d4e-4106-a86c-c59412b989c9" (UID: "0c4a0c21-3d4e-4106-a86c-c59412b989c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:54:24.724273 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.724256 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-maas-api-tls" pod "0c4a0c21-3d4e-4106-a86c-c59412b989c9" (UID: "0c4a0c21-3d4e-4106-a86c-c59412b989c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:54:24.751419 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.751385 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:24.823465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.823383 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:24.823465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.823409 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smftn\" (UniqueName: \"kubernetes.io/projected/0c4a0c21-3d4e-4106-a86c-c59412b989c9-kube-api-access-smftn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:24.823465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.823419 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c4a0c21-3d4e-4106-a86c-c59412b989c9-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:24.823465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.823430 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4a0c21-3d4e-4106-a86c-c59412b989c9-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:24.977180 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.977148 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:24.986089 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.986054 5591 generic.go:356] "Generic (PLEG): container finished" podID="0c4a0c21-3d4e-4106-a86c-c59412b989c9" containerID="c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921" exitCode=0 Nov 02 13:54:24.986229 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.986097 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" Nov 02 13:54:24.986229 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.986110 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" event={"ID":"0c4a0c21-3d4e-4106-a86c-c59412b989c9","Type":"ContainerDied","Data":"c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921"} Nov 02 13:54:24.986229 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.986158 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5" event={"ID":"0c4a0c21-3d4e-4106-a86c-c59412b989c9","Type":"ContainerDied","Data":"233281493aae6191a6e5f4dd58c7c3fa4cf4de2704e7c1c3b4f7539c40a29d1d"} Nov 02 13:54:24.986229 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.986205 5591 scope.go:122] "RemoveContainer" containerID="c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921" Nov 02 13:54:24.995635 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.995617 5591 scope.go:122] "RemoveContainer" containerID="c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921" Nov 02 13:54:24.995909 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:24.995887 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921\": container with ID starting with c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921 not found: ID does not exist" containerID="c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921" Nov 02 13:54:24.995949 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:24.995919 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921"} err="failed to get container status \"c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921\": rpc error: code = NotFound desc = could not find container \"c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921\": container with ID starting with c731e88f1d4300b71f3d3ae64e1dd75e048a396bd04b6f484f7be7f0f340d921 not found: ID does not exist" Nov 02 13:54:25.057458 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.057425 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5"] Nov 02 13:54:25.060492 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.060470 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6232c163-646b88d88d-lblx5"] Nov 02 13:54:25.174871 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.174835 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:25.429688 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.429573 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4a0c21-3d4e-4106-a86c-c59412b989c9" path="/var/lib/kubelet/pods/0c4a0c21-3d4e-4106-a86c-c59412b989c9/volumes" Nov 02 13:54:25.430095 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.430055 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9ae06c-3784-4b4d-93e6-dc71dadf11bb" path="/var/lib/kubelet/pods/2d9ae06c-3784-4b4d-93e6-dc71dadf11bb/volumes" Nov 02 13:54:25.436815 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.436792 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr"] Nov 02 13:54:25.437022 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.437001 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" containerID="cri-o://966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74" gracePeriod=30 Nov 02 13:54:25.530351 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.530288 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:25.535879 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.534078 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:54:25.736206 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.736129 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:25.943130 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:25.943101 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:26.108593 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.108514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:26.300936 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.300564 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:26.513123 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.513095 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:26.681946 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.681910 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr"] Nov 02 13:54:26.682429 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682408 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2d9ae06c-3784-4b4d-93e6-dc71dadf11bb" containerName="payload-pre-processing" Nov 02 13:54:26.682528 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682430 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9ae06c-3784-4b4d-93e6-dc71dadf11bb" containerName="payload-pre-processing" Nov 02 13:54:26.682528 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682443 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0c4a0c21-3d4e-4106-a86c-c59412b989c9" containerName="maas-api" Nov 02 13:54:26.682528 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682453 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4a0c21-3d4e-4106-a86c-c59412b989c9" containerName="maas-api" Nov 02 13:54:26.682528 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682468 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bfe9095b-b3c3-45b4-aa20-7b24fdf6af57" containerName="payload-pre-processing" Nov 02 13:54:26.682528 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682475 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe9095b-b3c3-45b4-aa20-7b24fdf6af57" containerName="payload-pre-processing" Nov 02 13:54:26.682528 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682509 5591 cpu_manager.go:414] "containerMap: removing container" podUID="33981cdf-9fc8-45a1-af5d-cc327b216099" containerName="payload-processing" Nov 02 13:54:26.682528 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682516 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33981cdf-9fc8-45a1-af5d-cc327b216099" containerName="payload-processing" Nov 02 13:54:26.682855 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682611 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9ae06c-3784-4b4d-93e6-dc71dadf11bb" containerName="payload-pre-processing" Nov 02 13:54:26.682855 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682624 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4a0c21-3d4e-4106-a86c-c59412b989c9" containerName="maas-api" Nov 02 13:54:26.682855 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682636 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe9095b-b3c3-45b4-aa20-7b24fdf6af57" containerName="payload-pre-processing" Nov 02 13:54:26.682855 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.682650 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="33981cdf-9fc8-45a1-af5d-cc327b216099" containerName="payload-processing" Nov 02 13:54:26.684856 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.684837 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.700606 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.700575 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f30173-openshift-default-dockercfg-tw6wb\"" Nov 02 13:54:26.714890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.714858 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr"] Nov 02 13:54:26.743000 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.742944 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.743000 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.742978 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.743519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.743017 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.743519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.743202 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.743519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.743312 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.743519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.743358 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjqd\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-kube-api-access-mhjqd\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.743519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.743415 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.743519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.743434 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.743519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.743479 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.753443 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.753414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:26.815299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.815221 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz"] Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845363 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845434 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845487 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845570 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845640 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845678 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjqd\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-kube-api-access-mhjqd\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845804 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845836 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.845859 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.846738 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.847028 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.847550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.847289 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.851905 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.850351 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.851905 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.850634 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.852137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.851923 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.853083 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.853058 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.878385 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.878312 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.878385 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.878342 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjqd\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-kube-api-access-mhjqd\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:26.923437 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.923403 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:26.998306 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:26.998275 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:27.163092 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.163038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:27.324686 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.324660 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr"] Nov 02 13:54:27.325964 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:27.325937 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34932ab0_cb39_43e2_b9e1_ce8cb25b5e57.slice/crio-6ffab15442717c10900702b471c9ca4c0d7d4b4091648b402f0079d45fa8a51c WatchSource:0}: Error finding container 6ffab15442717c10900702b471c9ca4c0d7d4b4091648b402f0079d45fa8a51c: Status 404 returned error can't find the container with id 6ffab15442717c10900702b471c9ca4c0d7d4b4091648b402f0079d45fa8a51c Nov 02 13:54:27.328058 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.328028 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:27.328153 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.328096 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:27.328153 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.328125 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:27.363999 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.363955 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7"] Nov 02 13:54:27.364213 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.364188 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" podUID="2bf283da-d37b-4518-b1da-dbe242c510e7" containerName="maas-api" containerID="cri-o://655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531" gracePeriod=30 Nov 02 13:54:27.376648 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.376400 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:27.616399 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.616374 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:27.756014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.755240 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-metrics-certs\") pod \"2bf283da-d37b-4518-b1da-dbe242c510e7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " Nov 02 13:54:27.756014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.755302 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf283da-d37b-4518-b1da-dbe242c510e7-kube-api-access-kgn84\" (UniqueName: \"kubernetes.io/projected/2bf283da-d37b-4518-b1da-dbe242c510e7-kube-api-access-kgn84\") pod \"2bf283da-d37b-4518-b1da-dbe242c510e7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " Nov 02 13:54:27.756014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.755719 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf283da-d37b-4518-b1da-dbe242c510e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf283da-d37b-4518-b1da-dbe242c510e7-openshift-service-ca\") pod \"2bf283da-d37b-4518-b1da-dbe242c510e7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " Nov 02 13:54:27.756014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.755847 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls\") pod \"2bf283da-d37b-4518-b1da-dbe242c510e7\" (UID: \"2bf283da-d37b-4518-b1da-dbe242c510e7\") " Nov 02 13:54:27.756692 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.756402 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf283da-d37b-4518-b1da-dbe242c510e7-openshift-service-ca" pod "2bf283da-d37b-4518-b1da-dbe242c510e7" (UID: "2bf283da-d37b-4518-b1da-dbe242c510e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:27.757619 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.757592 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-metrics-certs" pod "2bf283da-d37b-4518-b1da-dbe242c510e7" (UID: "2bf283da-d37b-4518-b1da-dbe242c510e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:54:27.758335 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.758307 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls" pod "2bf283da-d37b-4518-b1da-dbe242c510e7" (UID: "2bf283da-d37b-4518-b1da-dbe242c510e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:54:27.758335 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.758317 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf283da-d37b-4518-b1da-dbe242c510e7-kube-api-access-kgn84" pod "2bf283da-d37b-4518-b1da-dbe242c510e7" (UID: "2bf283da-d37b-4518-b1da-dbe242c510e7"). InnerVolumeSpecName "kube-api-access-kgn84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:27.758917 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.758894 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh"] Nov 02 13:54:27.759363 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.759347 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2bf283da-d37b-4518-b1da-dbe242c510e7" containerName="maas-api" Nov 02 13:54:27.759435 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.759364 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf283da-d37b-4518-b1da-dbe242c510e7" containerName="maas-api" Nov 02 13:54:27.759508 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.759483 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf283da-d37b-4518-b1da-dbe242c510e7" containerName="maas-api" Nov 02 13:54:27.762064 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.762043 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.769821 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.769726 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:27.784794 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.784771 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ad42e0-openshift-default-dockercfg-gmvz9\"" Nov 02 13:54:27.793604 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.793575 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh"] Nov 02 13:54:27.857035 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.856995 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857207 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857088 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857207 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857163 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857309 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857245 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqk6d\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-kube-api-access-qqk6d\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857348 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857304 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857354 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857437 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857386 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857520 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857446 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857520 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857488 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.857693 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857674 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf283da-d37b-4518-b1da-dbe242c510e7-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:27.857732 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857697 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:27.857732 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857709 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bf283da-d37b-4518-b1da-dbe242c510e7-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:27.857732 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.857717 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgn84\" (UniqueName: \"kubernetes.io/projected/2bf283da-d37b-4518-b1da-dbe242c510e7-kube-api-access-kgn84\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.958604 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.958704 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.958734 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.958771 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.960547 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.960938 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqk6d\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-kube-api-access-qqk6d\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.961062 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.961118 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.961154 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.961216 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.962176 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.962834 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.963375 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.963708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.963640 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.965438 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.965285 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:27.969085 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:27.969032 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:28.012076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.011601 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:28.012957 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.012899 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqk6d\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-kube-api-access-qqk6d\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:28.015826 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.015794 5591 generic.go:356] "Generic (PLEG): container finished" podID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerID="39bd8acae5b8ae2177aad45df4e0f097031733c12c98c90ce8c7b4e62895bfc0" exitCode=0 Nov 02 13:54:28.015939 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.015877 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" event={"ID":"0cb9de3e-bd88-4952-89a5-32692d236a0f","Type":"ContainerDied","Data":"39bd8acae5b8ae2177aad45df4e0f097031733c12c98c90ce8c7b4e62895bfc0"} Nov 02 13:54:28.016029 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.015954 5591 scope.go:122] "RemoveContainer" containerID="9d7440f4121f6351607709f1bbd92bbfc0ac3189c069a28b19d27e09d154f06f" Nov 02 13:54:28.017395 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.017371 5591 generic.go:356] "Generic (PLEG): container finished" podID="2bf283da-d37b-4518-b1da-dbe242c510e7" containerID="655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531" exitCode=0 Nov 02 13:54:28.017470 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.017453 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" event={"ID":"2bf283da-d37b-4518-b1da-dbe242c510e7","Type":"ContainerDied","Data":"655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531"} Nov 02 13:54:28.017542 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.017505 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" event={"ID":"2bf283da-d37b-4518-b1da-dbe242c510e7","Type":"ContainerDied","Data":"1454788eee21c5053326fd9b9acda2983a69be9e6d79919a9fe5d9a9a9c40913"} Nov 02 13:54:28.017619 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.017607 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7" Nov 02 13:54:28.022468 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.022370 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" event={"ID":"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57","Type":"ContainerStarted","Data":"4099ca9e33f9c33e61dcc098d110a5494783071df09f4a113933c8aeeb589219"} Nov 02 13:54:28.022468 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.022432 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" event={"ID":"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57","Type":"ContainerStarted","Data":"6ffab15442717c10900702b471c9ca4c0d7d4b4091648b402f0079d45fa8a51c"} Nov 02 13:54:28.029713 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.029696 5591 scope.go:122] "RemoveContainer" containerID="655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531" Nov 02 13:54:28.041538 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.041518 5591 scope.go:122] "RemoveContainer" containerID="655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531" Nov 02 13:54:28.041831 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:28.041811 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531\": container with ID starting with 655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531 not found: ID does not exist" containerID="655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531" Nov 02 13:54:28.041898 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.041838 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531"} err="failed to get container status \"655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531\": rpc error: code = NotFound desc = could not find container \"655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531\": container with ID starting with 655118405f018847c0efe215e7638ea6ccdf29e172aeb8eff17a412a414af531 not found: ID does not exist" Nov 02 13:54:28.065599 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.065563 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:28.074604 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.074566 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:28.095897 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.095862 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7"] Nov 02 13:54:28.112635 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.112592 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7818088-7bc99c9d95-pd6s7"] Nov 02 13:54:28.144505 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.144481 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:54:28.170143 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.168946 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" podStartSLOduration=2.168926907 podStartE2EDuration="2.168926907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:28.16756013 +0000 UTC m=+616.893181899" watchObservedRunningTime="2026-11-02 13:54:28.168926907 +0000 UTC m=+616.894548677" Nov 02 13:54:28.267354 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267264 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-socket\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267354 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267319 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-credential-socket\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267354 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267359 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb9de3e-bd88-4952-89a5-32692d236a0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb9de3e-bd88-4952-89a5-32692d236a0f-istiod-ca-cert\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267386 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-data\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267454 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-token\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-token\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267516 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-certs\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267550 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-socket" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:28.267629 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267616 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-kube-api-access-k7w2b\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-kube-api-access-k7w2b\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267891 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267669 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-podinfo\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267891 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267679 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-credential-socket" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:28.267891 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267709 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-envoy\") pod \"0cb9de3e-bd88-4952-89a5-32692d236a0f\" (UID: \"0cb9de3e-bd88-4952-89a5-32692d236a0f\") " Nov 02 13:54:28.267891 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267766 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-data" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:28.267891 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267795 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-certs" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:28.267891 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.267801 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb9de3e-bd88-4952-89a5-32692d236a0f-istiod-ca-cert" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:28.268166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.268068 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.268166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.268083 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.268166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.268095 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb9de3e-bd88-4952-89a5-32692d236a0f-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.268166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.268106 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.268166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.268114 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.269805 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.269778 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-envoy" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:28.269895 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.269804 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-token" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:28.270131 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.270114 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-podinfo" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:54:28.270173 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.270144 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-kube-api-access-k7w2b" pod "0cb9de3e-bd88-4952-89a5-32692d236a0f" (UID: "0cb9de3e-bd88-4952-89a5-32692d236a0f"). InnerVolumeSpecName "kube-api-access-k7w2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:28.291065 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.288666 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:28.368822 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.368791 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.368822 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.368821 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7w2b\" (UniqueName: \"kubernetes.io/projected/0cb9de3e-bd88-4952-89a5-32692d236a0f-kube-api-access-k7w2b\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.368822 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.368831 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.369056 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.368840 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb9de3e-bd88-4952-89a5-32692d236a0f-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:28.413613 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.413581 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh"] Nov 02 13:54:28.416571 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:28.416542 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7d3fe4_0505_4a74_a801_90cd65ea1a41.slice/crio-533b8f2c32b408473d201a8bd6431245fa28c9f1a40bd7d4f8b2380dec831880 WatchSource:0}: Error finding container 533b8f2c32b408473d201a8bd6431245fa28c9f1a40bd7d4f8b2380dec831880: Status 404 returned error can't find the container with id 533b8f2c32b408473d201a8bd6431245fa28c9f1a40bd7d4f8b2380dec831880 Nov 02 13:54:28.418913 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.418881 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:28.419040 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.418953 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:28.419040 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.419013 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:28.623957 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.623851 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" podUID="752cca65-73af-43de-9959-36c6bcf2b750" containerName="payload-processing" containerID="cri-o://fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c" gracePeriod=25 Nov 02 13:54:28.662675 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.662621 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" podUID="78cccbb2-d292-47bf-be37-4b3c244054a8" containerName="payload-pre-processing" containerID="cri-o://776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578" gracePeriod=25 Nov 02 13:54:28.695892 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:28.695865 5591 logging.go:55] [core] [Channel #146 SubChannel #148]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:50052", ServerName: "10.133.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:50052: connect: connection refused" Nov 02 13:54:28.696055 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:28.695903 5591 logging.go:55] [core] [Channel #147 SubChannel #149]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.37:50052", ServerName: "10.133.0.37:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.37:50052: connect: connection refused" Nov 02 13:54:28.906604 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.906577 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:54:28.942219 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.942194 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:54:28.973138 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.973107 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752cca65-73af-43de-9959-36c6bcf2b750-kube-api-access-dm4jd\" (UniqueName: \"kubernetes.io/projected/752cca65-73af-43de-9959-36c6bcf2b750-kube-api-access-dm4jd\") pod \"752cca65-73af-43de-9959-36c6bcf2b750\" (UID: \"752cca65-73af-43de-9959-36c6bcf2b750\") " Nov 02 13:54:28.973276 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.973254 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752cca65-73af-43de-9959-36c6bcf2b750-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cca65-73af-43de-9959-36c6bcf2b750-plugins-config-volume\") pod \"752cca65-73af-43de-9959-36c6bcf2b750\" (UID: \"752cca65-73af-43de-9959-36c6bcf2b750\") " Nov 02 13:54:28.973607 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.973582 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752cca65-73af-43de-9959-36c6bcf2b750-plugins-config-volume" pod "752cca65-73af-43de-9959-36c6bcf2b750" (UID: "752cca65-73af-43de-9959-36c6bcf2b750"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:28.975125 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.975102 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752cca65-73af-43de-9959-36c6bcf2b750-kube-api-access-dm4jd" pod "752cca65-73af-43de-9959-36c6bcf2b750" (UID: "752cca65-73af-43de-9959-36c6bcf2b750"). InnerVolumeSpecName "kube-api-access-dm4jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:28.999024 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:28.998973 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:29.004127 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.004102 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:29.028509 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.028478 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" event={"ID":"0cb9de3e-bd88-4952-89a5-32692d236a0f","Type":"ContainerDied","Data":"5d4070d52ba44251641d8468ca49843acb1e8925fe2c19b01dfd2fbedd467161"} Nov 02 13:54:29.028509 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.028495 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk" Nov 02 13:54:29.028720 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.028533 5591 scope.go:122] "RemoveContainer" containerID="39bd8acae5b8ae2177aad45df4e0f097031733c12c98c90ce8c7b4e62895bfc0" Nov 02 13:54:29.029967 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.029943 5591 generic.go:356] "Generic (PLEG): container finished" podID="752cca65-73af-43de-9959-36c6bcf2b750" containerID="fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c" exitCode=0 Nov 02 13:54:29.030088 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.030016 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" Nov 02 13:54:29.030088 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.030021 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" event={"ID":"752cca65-73af-43de-9959-36c6bcf2b750","Type":"ContainerDied","Data":"fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c"} Nov 02 13:54:29.030088 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.030079 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" event={"ID":"752cca65-73af-43de-9959-36c6bcf2b750","Type":"ContainerDied","Data":"1f4197540154154e59b6afb8d28ff34783ff14c2f509d5928a082ddee8398db3"} Nov 02 13:54:29.032014 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.031965 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" event={"ID":"8e7d3fe4-0505-4a74-a801-90cd65ea1a41","Type":"ContainerStarted","Data":"fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03"} Nov 02 13:54:29.032262 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.032040 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" event={"ID":"8e7d3fe4-0505-4a74-a801-90cd65ea1a41","Type":"ContainerStarted","Data":"533b8f2c32b408473d201a8bd6431245fa28c9f1a40bd7d4f8b2380dec831880"} Nov 02 13:54:29.034677 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.034649 5591 generic.go:356] "Generic (PLEG): container finished" podID="78cccbb2-d292-47bf-be37-4b3c244054a8" containerID="776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578" exitCode=0 Nov 02 13:54:29.034786 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.034706 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" Nov 02 13:54:29.034786 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.034728 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" event={"ID":"78cccbb2-d292-47bf-be37-4b3c244054a8","Type":"ContainerDied","Data":"776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578"} Nov 02 13:54:29.034786 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.034777 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" event={"ID":"78cccbb2-d292-47bf-be37-4b3c244054a8","Type":"ContainerDied","Data":"ce5a1b07cebc2c982a167954702a077f2f2a10a2b701129bafbef8e5d31a4697"} Nov 02 13:54:29.035031 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.035011 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:29.036015 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.035972 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:29.038727 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.038705 5591 scope.go:122] "RemoveContainer" containerID="fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c" Nov 02 13:54:29.047195 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.047180 5591 scope.go:122] "RemoveContainer" containerID="fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c" Nov 02 13:54:29.047537 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:29.047510 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c\": container with ID starting with fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c not found: ID does not exist" containerID="fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c" Nov 02 13:54:29.047624 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.047548 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c"} err="failed to get container status \"fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c\": rpc error: code = NotFound desc = could not find container \"fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c\": container with ID starting with fd00826708832a7430bfbdcec10e63a041cf18297e6da0aa1d131121c305901c not found: ID does not exist" Nov 02 13:54:29.047624 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.047571 5591 scope.go:122] "RemoveContainer" containerID="776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578" Nov 02 13:54:29.055518 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.055502 5591 scope.go:122] "RemoveContainer" containerID="776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578" Nov 02 13:54:29.055807 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:29.055790 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578\": container with ID starting with 776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578 not found: ID does not exist" containerID="776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578" Nov 02 13:54:29.055862 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.055815 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578"} err="failed to get container status \"776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578\": rpc error: code = NotFound desc = could not find container \"776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578\": container with ID starting with 776a20a71bc71beb3a5eadc30c3e35db01bf80d8e23ca8c12684ff7dc5723578 not found: ID does not exist" Nov 02 13:54:29.074121 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.074093 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cccbb2-d292-47bf-be37-4b3c244054a8-kube-api-access-qsp5t\" (UniqueName: \"kubernetes.io/projected/78cccbb2-d292-47bf-be37-4b3c244054a8-kube-api-access-qsp5t\") pod \"78cccbb2-d292-47bf-be37-4b3c244054a8\" (UID: \"78cccbb2-d292-47bf-be37-4b3c244054a8\") " Nov 02 13:54:29.074264 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.074158 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cccbb2-d292-47bf-be37-4b3c244054a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cccbb2-d292-47bf-be37-4b3c244054a8-plugins-config-volume\") pod \"78cccbb2-d292-47bf-be37-4b3c244054a8\" (UID: \"78cccbb2-d292-47bf-be37-4b3c244054a8\") " Nov 02 13:54:29.074418 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.074404 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cca65-73af-43de-9959-36c6bcf2b750-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:29.074490 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.074418 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4jd\" (UniqueName: \"kubernetes.io/projected/752cca65-73af-43de-9959-36c6bcf2b750-kube-api-access-dm4jd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:29.074541 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.074505 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cccbb2-d292-47bf-be37-4b3c244054a8-plugins-config-volume" pod "78cccbb2-d292-47bf-be37-4b3c244054a8" (UID: "78cccbb2-d292-47bf-be37-4b3c244054a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:29.074841 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.074827 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:29.076298 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.076273 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cccbb2-d292-47bf-be37-4b3c244054a8-kube-api-access-qsp5t" pod "78cccbb2-d292-47bf-be37-4b3c244054a8" (UID: "78cccbb2-d292-47bf-be37-4b3c244054a8"). InnerVolumeSpecName "kube-api-access-qsp5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:29.080087 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.080067 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:29.088820 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.088785 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z"] Nov 02 13:54:29.092377 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.092356 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z"] Nov 02 13:54:29.177206 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.177101 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cccbb2-d292-47bf-be37-4b3c244054a8-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:29.177206 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.177137 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsp5t\" (UniqueName: \"kubernetes.io/projected/78cccbb2-d292-47bf-be37-4b3c244054a8-kube-api-access-qsp5t\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:29.181099 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.181072 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk"] Nov 02 13:54:29.184551 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.184524 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ebfc3db5-openshift-default-6c57d7d657-ksjxk"] Nov 02 13:54:29.231381 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.231331 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" podStartSLOduration=2.231318807 podStartE2EDuration="2.231318807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:29.229732774 +0000 UTC m=+617.955354545" watchObservedRunningTime="2026-11-02 13:54:29.231318807 +0000 UTC m=+617.956940555" Nov 02 13:54:29.383308 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.383277 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw"] Nov 02 13:54:29.392432 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.392405 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw"] Nov 02 13:54:29.429041 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.428951 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" path="/var/lib/kubelet/pods/0cb9de3e-bd88-4952-89a5-32692d236a0f/volumes" Nov 02 13:54:29.429405 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.429390 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf283da-d37b-4518-b1da-dbe242c510e7" path="/var/lib/kubelet/pods/2bf283da-d37b-4518-b1da-dbe242c510e7/volumes" Nov 02 13:54:29.429718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.429706 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752cca65-73af-43de-9959-36c6bcf2b750" path="/var/lib/kubelet/pods/752cca65-73af-43de-9959-36c6bcf2b750/volumes" Nov 02 13:54:29.430397 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.430381 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cccbb2-d292-47bf-be37-4b3c244054a8" path="/var/lib/kubelet/pods/78cccbb2-d292-47bf-be37-4b3c244054a8/volumes" Nov 02 13:54:29.454664 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.454634 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:29.566623 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.566589 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm"] Nov 02 13:54:29.567009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.566996 5591 cpu_manager.go:414] "containerMap: removing container" podUID="78cccbb2-d292-47bf-be37-4b3c244054a8" containerName="payload-pre-processing" Nov 02 13:54:29.567076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567011 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cccbb2-d292-47bf-be37-4b3c244054a8" containerName="payload-pre-processing" Nov 02 13:54:29.567076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567025 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" Nov 02 13:54:29.567076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567029 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" Nov 02 13:54:29.567076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567038 5591 cpu_manager.go:414] "containerMap: removing container" podUID="752cca65-73af-43de-9959-36c6bcf2b750" containerName="payload-processing" Nov 02 13:54:29.567076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567045 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752cca65-73af-43de-9959-36c6bcf2b750" containerName="payload-processing" Nov 02 13:54:29.567076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567057 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" Nov 02 13:54:29.567076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567062 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" Nov 02 13:54:29.567282 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567121 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cccbb2-d292-47bf-be37-4b3c244054a8" containerName="payload-pre-processing" Nov 02 13:54:29.567282 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567129 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" Nov 02 13:54:29.567282 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567134 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="752cca65-73af-43de-9959-36c6bcf2b750" containerName="payload-processing" Nov 02 13:54:29.567282 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.567140 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb9de3e-bd88-4952-89a5-32692d236a0f" containerName="istio-proxy" Nov 02 13:54:29.570377 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.570360 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:29.589318 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.589290 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1a562c\"" Nov 02 13:54:29.589442 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.589317 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1a562c-dockercfg-74h82\"" Nov 02 13:54:29.600368 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.600341 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm"] Nov 02 13:54:29.653948 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.650878 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr"] Nov 02 13:54:29.679236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.679147 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:29.682113 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.682079 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vsg5\" (UniqueName: \"kubernetes.io/projected/5fe100e2-966f-4c81-b6d6-41347f79710f-kube-api-access-6vsg5\") pod \"payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm\" (UID: \"5fe100e2-966f-4c81-b6d6-41347f79710f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:29.682327 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.682308 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe100e2-966f-4c81-b6d6-41347f79710f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm\" (UID: \"5fe100e2-966f-4c81-b6d6-41347f79710f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:29.690788 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.690723 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr"] Nov 02 13:54:29.690971 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.690864 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:29.696057 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.695899 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-c57ba6aa-7cb854f5bc-vb7bw" podUID="78cccbb2-d292-47bf-be37-4b3c244054a8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.37:50052\" within 1s: context deadline exceeded" Nov 02 13:54:29.701013 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.696459 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-c57ba6aa-68dcf5cff-6f77z" podUID="752cca65-73af-43de-9959-36c6bcf2b750" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.38:50052\" within 1s: context deadline exceeded" Nov 02 13:54:29.782951 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.782919 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vsg5\" (UniqueName: \"kubernetes.io/projected/5fe100e2-966f-4c81-b6d6-41347f79710f-kube-api-access-6vsg5\") pod \"payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm\" (UID: \"5fe100e2-966f-4c81-b6d6-41347f79710f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:29.783123 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.783020 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe100e2-966f-4c81-b6d6-41347f79710f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm\" (UID: \"5fe100e2-966f-4c81-b6d6-41347f79710f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:29.783123 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.783058 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr\" (UID: \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:29.783123 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.783109 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj76f\" (UniqueName: \"kubernetes.io/projected/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-kube-api-access-dj76f\") pod \"payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr\" (UID: \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:29.783749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.783729 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe100e2-966f-4c81-b6d6-41347f79710f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm\" (UID: \"5fe100e2-966f-4c81-b6d6-41347f79710f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:29.806966 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.806935 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vsg5\" (UniqueName: \"kubernetes.io/projected/5fe100e2-966f-4c81-b6d6-41347f79710f-kube-api-access-6vsg5\") pod \"payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm\" (UID: \"5fe100e2-966f-4c81-b6d6-41347f79710f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:29.881678 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.881643 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:29.883764 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.883742 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr\" (UID: \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:29.883813 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.883792 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj76f\" (UniqueName: \"kubernetes.io/projected/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-kube-api-access-dj76f\") pod \"payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr\" (UID: \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:29.884469 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.884451 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr\" (UID: \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:29.910447 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:29.910416 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj76f\" (UniqueName: \"kubernetes.io/projected/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-kube-api-access-dj76f\") pod \"payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr\" (UID: \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:30.022364 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.022332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:30.028375 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.028346 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:30.055811 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.055780 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:30.056655 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.056631 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:54:30.145453 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.145419 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:54:30.194890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.194860 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm"] Nov 02 13:54:30.195904 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:30.195881 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe100e2_966f_4c81_b6d6_41347f79710f.slice/crio-b97b0652f182dd27b615a20dc9e4a0404bbc7671b14bc30ab9e14c0a82bb3ef0 WatchSource:0}: Error finding container b97b0652f182dd27b615a20dc9e4a0404bbc7671b14bc30ab9e14c0a82bb3ef0: Status 404 returned error can't find the container with id b97b0652f182dd27b615a20dc9e4a0404bbc7671b14bc30ab9e14c0a82bb3ef0 Nov 02 13:54:30.205086 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.205027 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:30.350092 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.350064 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr"] Nov 02 13:54:30.351304 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:30.351276 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe3bb4e_5a5c_4019_aada_a451a33c84c6.slice/crio-1742b9fc9a8cfad4e2eec5ec3da73c86802c339d49e200629f57cc28411881b4 WatchSource:0}: Error finding container 1742b9fc9a8cfad4e2eec5ec3da73c86802c339d49e200629f57cc28411881b4: Status 404 returned error can't find the container with id 1742b9fc9a8cfad4e2eec5ec3da73c86802c339d49e200629f57cc28411881b4 Nov 02 13:54:30.617880 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.617849 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:30.676374 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.676347 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:54:30.795209 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795129 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-envoy\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795209 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795168 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-podinfo\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795209 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795194 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-certs\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795457 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795300 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-socket\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795457 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795409 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-credential-socket\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795563 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795457 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istiod-ca-cert\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795563 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795493 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-token\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-token\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795563 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795520 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-kube-api-access-qx8nd\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-kube-api-access-qx8nd\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795563 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795520 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-certs" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:30.795760 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795560 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-socket" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:30.795760 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795573 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-data\") pod \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\" (UID: \"1baef57a-a73f-40bc-9acc-e3f13ad84a1a\") " Nov 02 13:54:30.795760 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795697 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-credential-socket" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:30.795900 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795782 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istiod-ca-cert" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:30.796052 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795953 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:30.796052 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.795975 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:30.796052 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.796022 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:30.796052 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.796036 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:30.796286 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.796162 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-data" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:30.797439 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.797418 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-envoy" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:54:30.797599 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.797581 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-token" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:30.797802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.797789 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-kube-api-access-qx8nd" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "kube-api-access-qx8nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:30.797838 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.797805 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-podinfo" pod "1baef57a-a73f-40bc-9acc-e3f13ad84a1a" (UID: "1baef57a-a73f-40bc-9acc-e3f13ad84a1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:54:30.860732 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.860693 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:30.896880 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.896842 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:30.896880 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.896874 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:30.896880 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.896883 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:30.897131 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.896891 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:30.897131 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:30.896901 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8nd\" (UniqueName: \"kubernetes.io/projected/1baef57a-a73f-40bc-9acc-e3f13ad84a1a-kube-api-access-qx8nd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:31.058674 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.058574 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:31.062601 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.062571 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" event={"ID":"9fe3bb4e-5a5c-4019-aada-a451a33c84c6","Type":"ContainerStarted","Data":"d6976699aaaedc4b4aa3013bd6461cf40e35cad2b3614f68991f428fa24559ea"} Nov 02 13:54:31.062764 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.062635 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" event={"ID":"9fe3bb4e-5a5c-4019-aada-a451a33c84c6","Type":"ContainerStarted","Data":"1742b9fc9a8cfad4e2eec5ec3da73c86802c339d49e200629f57cc28411881b4"} Nov 02 13:54:31.062764 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.062689 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:31.064124 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.064092 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" event={"ID":"5fe100e2-966f-4c81-b6d6-41347f79710f","Type":"ContainerStarted","Data":"825ce3a2dd8f072fb5f41b3131f5888138a2f2e5e9a5c598321f78364c966d13"} Nov 02 13:54:31.064273 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.064144 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" event={"ID":"5fe100e2-966f-4c81-b6d6-41347f79710f","Type":"ContainerStarted","Data":"b97b0652f182dd27b615a20dc9e4a0404bbc7671b14bc30ab9e14c0a82bb3ef0"} Nov 02 13:54:31.064273 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.064186 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:31.065909 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.065887 5591 generic.go:356] "Generic (PLEG): container finished" podID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerID="966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74" exitCode=0 Nov 02 13:54:31.066047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.065931 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" Nov 02 13:54:31.066047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.065916 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" event={"ID":"1baef57a-a73f-40bc-9acc-e3f13ad84a1a","Type":"ContainerDied","Data":"966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74"} Nov 02 13:54:31.066047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.066029 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr" event={"ID":"1baef57a-a73f-40bc-9acc-e3f13ad84a1a","Type":"ContainerDied","Data":"0cc1d9973dd2cc81a0750ac9b641231e959b1323335b5ec2947f227bdff85f33"} Nov 02 13:54:31.066219 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.066053 5591 scope.go:122] "RemoveContainer" containerID="966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74" Nov 02 13:54:31.075855 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:31.075828 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:54:31.076042 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.076027 5591 scope.go:122] "RemoveContainer" containerID="557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9" Nov 02 13:54:31.083807 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.083793 5591 scope.go:122] "RemoveContainer" containerID="966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74" Nov 02 13:54:31.084101 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:31.084083 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74\": container with ID starting with 966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74 not found: ID does not exist" containerID="966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74" Nov 02 13:54:31.084139 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.084108 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74"} err="failed to get container status \"966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74\": rpc error: code = NotFound desc = could not find container \"966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74\": container with ID starting with 966229f28638ee7fe960994426fd8ab49921f09192dd23568512b901be846a74 not found: ID does not exist" Nov 02 13:54:31.084139 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.084135 5591 scope.go:122] "RemoveContainer" containerID="557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9" Nov 02 13:54:31.084387 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:31.084371 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9\": container with ID starting with 557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9 not found: ID does not exist" containerID="557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9" Nov 02 13:54:31.084428 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.084393 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9"} err="failed to get container status \"557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9\": rpc error: code = NotFound desc = could not find container \"557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9\": container with ID starting with 557b9372c36f4cd34b8f617e974054d2a2bb15eb698d63b2dd1f4bbf419c95a9 not found: ID does not exist" Nov 02 13:54:31.112185 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.112141 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" podStartSLOduration=2.112130204 podStartE2EDuration="2.112130204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:31.111236723 +0000 UTC m=+619.836858490" watchObservedRunningTime="2026-11-02 13:54:31.112130204 +0000 UTC m=+619.837751979" Nov 02 13:54:31.150378 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.150340 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr"] Nov 02 13:54:31.158454 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.158425 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6232c163-gw-openshift-default-765f7d95bf-84mwr"] Nov 02 13:54:31.212646 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.212589 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" podStartSLOduration=2.212577471 podStartE2EDuration="2.212577471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:31.211944593 +0000 UTC m=+619.937566363" watchObservedRunningTime="2026-11-02 13:54:31.212577471 +0000 UTC m=+619.938199272" Nov 02 13:54:31.429083 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.429053 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" path="/var/lib/kubelet/pods/1baef57a-a73f-40bc-9acc-e3f13ad84a1a/volumes" Nov 02 13:54:31.815710 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:31.815608 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" podUID="b3b7639b-4680-4d6f-b679-7a41f28f9277" containerName="payload-pre-processing" containerID="cri-o://051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356" gracePeriod=25 Nov 02 13:54:32.062908 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.062875 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:54:32.079917 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.079828 5591 generic.go:356] "Generic (PLEG): container finished" podID="b3b7639b-4680-4d6f-b679-7a41f28f9277" containerID="051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356" exitCode=0 Nov 02 13:54:32.079917 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.079877 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" Nov 02 13:54:32.080120 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.079930 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" event={"ID":"b3b7639b-4680-4d6f-b679-7a41f28f9277","Type":"ContainerDied","Data":"051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356"} Nov 02 13:54:32.080120 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.080017 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz" event={"ID":"b3b7639b-4680-4d6f-b679-7a41f28f9277","Type":"ContainerDied","Data":"e5fe260dc56781316e53857f4d279dd5c114b697f6fc59333b7e387e28af25aa"} Nov 02 13:54:32.080120 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.080041 5591 scope.go:122] "RemoveContainer" containerID="051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356" Nov 02 13:54:32.088444 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.088427 5591 scope.go:122] "RemoveContainer" containerID="051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356" Nov 02 13:54:32.088708 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:32.088688 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356\": container with ID starting with 051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356 not found: ID does not exist" containerID="051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356" Nov 02 13:54:32.088774 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.088715 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356"} err="failed to get container status \"051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356\": rpc error: code = NotFound desc = could not find container \"051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356\": container with ID starting with 051d919fb178254da5a169f6841cb3d1980c4358f82444c74309261e74cc6356 not found: ID does not exist" Nov 02 13:54:32.208079 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.208041 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b7639b-4680-4d6f-b679-7a41f28f9277-kube-api-access-866bh\" (UniqueName: \"kubernetes.io/projected/b3b7639b-4680-4d6f-b679-7a41f28f9277-kube-api-access-866bh\") pod \"b3b7639b-4680-4d6f-b679-7a41f28f9277\" (UID: \"b3b7639b-4680-4d6f-b679-7a41f28f9277\") " Nov 02 13:54:32.208249 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.208209 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b7639b-4680-4d6f-b679-7a41f28f9277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7639b-4680-4d6f-b679-7a41f28f9277-plugins-config-volume\") pod \"b3b7639b-4680-4d6f-b679-7a41f28f9277\" (UID: \"b3b7639b-4680-4d6f-b679-7a41f28f9277\") " Nov 02 13:54:32.208653 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.208621 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b7639b-4680-4d6f-b679-7a41f28f9277-plugins-config-volume" pod "b3b7639b-4680-4d6f-b679-7a41f28f9277" (UID: "b3b7639b-4680-4d6f-b679-7a41f28f9277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:54:32.210253 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.210227 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b7639b-4680-4d6f-b679-7a41f28f9277-kube-api-access-866bh" pod "b3b7639b-4680-4d6f-b679-7a41f28f9277" (UID: "b3b7639b-4680-4d6f-b679-7a41f28f9277"). InnerVolumeSpecName "kube-api-access-866bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:54:32.309624 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.309584 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-866bh\" (UniqueName: \"kubernetes.io/projected/b3b7639b-4680-4d6f-b679-7a41f28f9277-kube-api-access-866bh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:32.309624 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.309617 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7639b-4680-4d6f-b679-7a41f28f9277-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:54:32.438015 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.437963 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz"] Nov 02 13:54:32.445504 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.445473 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e7818088-85c7994b6c-65gcz"] Nov 02 13:54:32.483882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.483812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:32.655476 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655399 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp"] Nov 02 13:54:32.655820 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655807 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b3b7639b-4680-4d6f-b679-7a41f28f9277" containerName="payload-pre-processing" Nov 02 13:54:32.655865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655821 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b7639b-4680-4d6f-b679-7a41f28f9277" containerName="payload-pre-processing" Nov 02 13:54:32.655865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655836 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" Nov 02 13:54:32.655865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655841 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" Nov 02 13:54:32.655865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655848 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" Nov 02 13:54:32.655865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655853 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" Nov 02 13:54:32.656050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655908 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" Nov 02 13:54:32.656050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655916 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baef57a-a73f-40bc-9acc-e3f13ad84a1a" containerName="istio-proxy" Nov 02 13:54:32.656050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.655927 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b7639b-4680-4d6f-b679-7a41f28f9277" containerName="payload-pre-processing" Nov 02 13:54:32.658668 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.658650 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.728005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.713177 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp"] Nov 02 13:54:32.738314 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.738272 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:32.752350 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.752316 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 13:54:32.752640 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.752440 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 13:54:32.752640 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.752520 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-0969b1-serving-cert\"" Nov 02 13:54:32.752640 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.752542 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 13:54:32.752767 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.752666 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 13:54:32.813685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.813648 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c61935-488e-4e61-85d4-618c252f458f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.813844 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.813693 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.813844 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.813722 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-metrics-certs\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.813967 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.813857 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtx8\" (UniqueName: \"kubernetes.io/projected/34c61935-488e-4e61-85d4-618c252f458f-kube-api-access-2xtx8\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.914525 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.914429 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtx8\" (UniqueName: \"kubernetes.io/projected/34c61935-488e-4e61-85d4-618c252f458f-kube-api-access-2xtx8\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.914683 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.914567 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c61935-488e-4e61-85d4-618c252f458f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.914683 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.914612 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.914790 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.914707 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-metrics-certs\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.914790 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:32.914726 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-serving-cert: secret "maas-api-e2e-worker-w2-0969b1-serving-cert" not found Nov 02 13:54:32.914852 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:32.914790 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls podName:34c61935-488e-4e61-85d4-618c252f458f nodeName:}" failed. No retries permitted until 2026-11-02 13:54:33.414769333 +0000 UTC m=+622.140391082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls") pod "maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" (UID: "34c61935-488e-4e61-85d4-618c252f458f") : secret "maas-api-e2e-worker-w2-0969b1-serving-cert" not found Nov 02 13:54:32.915290 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.915260 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c61935-488e-4e61-85d4-618c252f458f-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.917093 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.917073 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-metrics-certs\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:32.919684 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.919662 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:32.942409 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:32.942384 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtx8\" (UniqueName: \"kubernetes.io/projected/34c61935-488e-4e61-85d4-618c252f458f-kube-api-access-2xtx8\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:33.109560 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:33.109525 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:33.343193 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:33.343111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:33.419767 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:33.419736 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:33.422166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:33.422143 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls\") pod \"maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:33.432750 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:33.432718 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b7639b-4680-4d6f-b679-7a41f28f9277" path="/var/lib/kubelet/pods/b3b7639b-4680-4d6f-b679-7a41f28f9277/volumes" Nov 02 13:54:33.570118 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:33.570087 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:33.898043 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:33.898016 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp"] Nov 02 13:54:33.899251 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:33.899228 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c61935_488e_4e61_85d4_618c252f458f.slice/crio-7a3b84afa21b2b7f5f95bc5dd9d78c883466d95adb5ed1fabfc7a5de29dc897c WatchSource:0}: Error finding container 7a3b84afa21b2b7f5f95bc5dd9d78c883466d95adb5ed1fabfc7a5de29dc897c: Status 404 returned error can't find the container with id 7a3b84afa21b2b7f5f95bc5dd9d78c883466d95adb5ed1fabfc7a5de29dc897c Nov 02 13:54:33.939361 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:33.939331 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:34.091511 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:34.091469 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" event={"ID":"34c61935-488e-4e61-85d4-618c252f458f","Type":"ContainerStarted","Data":"7a3b84afa21b2b7f5f95bc5dd9d78c883466d95adb5ed1fabfc7a5de29dc897c"} Nov 02 13:54:34.143694 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:34.143655 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:34.581225 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:34.581151 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:34.758952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:34.758903 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:34.887773 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:34.887737 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2"] Nov 02 13:54:34.890632 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:34.890616 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:34.911978 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:34.911955 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-f30173-serving-cert\"" Nov 02 13:54:34.919427 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:34.919396 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2"] Nov 02 13:54:35.036133 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.036097 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-metrics-certs\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.036289 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.036249 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78qp\" (UniqueName: \"kubernetes.io/projected/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-kube-api-access-l78qp\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.036362 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.036340 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.036410 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.036397 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.098012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.097964 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" event={"ID":"34c61935-488e-4e61-85d4-618c252f458f","Type":"ContainerStarted","Data":"6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d"} Nov 02 13:54:35.098169 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.098119 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:35.122971 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.122920 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:35.137181 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.137154 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-metrics-certs\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.137313 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.137298 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78qp\" (UniqueName: \"kubernetes.io/projected/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-kube-api-access-l78qp\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.137351 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.137337 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.137386 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.137379 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.137519 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:35.137504 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-serving-cert: secret "maas-api-e2e-worker-w3-f30173-serving-cert" not found Nov 02 13:54:35.137604 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:54:35.137585 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls podName:3bead4e2-f184-49b9-8af8-36c25b7bc7dc nodeName:}" failed. No retries permitted until 2026-11-02 13:54:35.637565129 +0000 UTC m=+624.363186890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls") pod "maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" (UID: "3bead4e2-f184-49b9-8af8-36c25b7bc7dc") : secret "maas-api-e2e-worker-w3-f30173-serving-cert" not found Nov 02 13:54:35.138130 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.138080 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.139804 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.139774 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-metrics-certs\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.176834 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.176788 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78qp\" (UniqueName: \"kubernetes.io/projected/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-kube-api-access-l78qp\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.438373 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.438273 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:35.633009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.629308 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" podStartSLOduration=3.3128353329999998 podStartE2EDuration="3.629286711s" totalImagesPullingTime="316.451378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:35.156398491 +0000 UTC m=+623.882020273" watchObservedRunningTime="2026-11-02 13:54:35.629286711 +0000 UTC m=+624.354908513" Nov 02 13:54:35.633009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.629680 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7"] Nov 02 13:54:35.645458 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.644929 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.654009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.650371 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:35.655190 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.655147 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.668260 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.668229 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f30173\"" Nov 02 13:54:35.668580 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.668561 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f30173-dockercfg-b26ww\"" Nov 02 13:54:35.681322 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.679495 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7"] Nov 02 13:54:35.695084 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.694951 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:35.747068 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.747022 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7\" (UID: \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:35.747242 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.747117 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjf7d\" (UniqueName: \"kubernetes.io/projected/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-kube-api-access-rjf7d\") pod \"payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7\" (UID: \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:35.803026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.802971 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:35.848613 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.848575 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7\" (UID: \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:35.848797 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.848633 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjf7d\" (UniqueName: \"kubernetes.io/projected/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-kube-api-access-rjf7d\") pod \"payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7\" (UID: \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:35.849360 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.849337 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7\" (UID: \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:35.873298 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.873269 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjf7d\" (UniqueName: \"kubernetes.io/projected/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-kube-api-access-rjf7d\") pod \"payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7\" (UID: \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:35.999420 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:35.999330 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:36.106433 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:36.106406 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2"] Nov 02 13:54:36.108064 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:36.108037 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bead4e2_f184_49b9_8af8_36c25b7bc7dc.slice/crio-f217fbaf8da0064a3bb1ccbd8e931b9af9c2ae0aafb1f099254f57c82edd76cc WatchSource:0}: Error finding container f217fbaf8da0064a3bb1ccbd8e931b9af9c2ae0aafb1f099254f57c82edd76cc: Status 404 returned error can't find the container with id f217fbaf8da0064a3bb1ccbd8e931b9af9c2ae0aafb1f099254f57c82edd76cc Nov 02 13:54:36.307850 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:36.307821 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7"] Nov 02 13:54:36.316031 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:36.310057 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f0d23b_ef8c_4f01_a66d_622d13eccd48.slice/crio-7b4e1edb049291c4f80354f037d0fa64e65671926f4618861d5b8f9748c7f82a WatchSource:0}: Error finding container 7b4e1edb049291c4f80354f037d0fa64e65671926f4618861d5b8f9748c7f82a: Status 404 returned error can't find the container with id 7b4e1edb049291c4f80354f037d0fa64e65671926f4618861d5b8f9748c7f82a Nov 02 13:54:36.465809 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:36.465777 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:36.634923 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:36.634875 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:37.109837 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.109743 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" event={"ID":"3bead4e2-f184-49b9-8af8-36c25b7bc7dc","Type":"ContainerStarted","Data":"fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721"} Nov 02 13:54:37.109837 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.109806 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" event={"ID":"3bead4e2-f184-49b9-8af8-36c25b7bc7dc","Type":"ContainerStarted","Data":"f217fbaf8da0064a3bb1ccbd8e931b9af9c2ae0aafb1f099254f57c82edd76cc"} Nov 02 13:54:37.110070 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.109940 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:37.111243 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.111219 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" event={"ID":"e6f0d23b-ef8c-4f01-a66d-622d13eccd48","Type":"ContainerStarted","Data":"ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953"} Nov 02 13:54:37.111351 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.111258 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" event={"ID":"e6f0d23b-ef8c-4f01-a66d-622d13eccd48","Type":"ContainerStarted","Data":"7b4e1edb049291c4f80354f037d0fa64e65671926f4618861d5b8f9748c7f82a"} Nov 02 13:54:37.111351 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.111301 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:37.154957 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.154902 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" podStartSLOduration=2.866665843 podStartE2EDuration="3.154888075s" totalImagesPullingTime="288.222232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:37.154468494 +0000 UTC m=+625.880090265" watchObservedRunningTime="2026-11-02 13:54:37.154888075 +0000 UTC m=+625.880509907" Nov 02 13:54:37.200291 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.200229 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" podStartSLOduration=2.20021537 podStartE2EDuration="2.20021537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:37.198382667 +0000 UTC m=+625.924004431" watchObservedRunningTime="2026-11-02 13:54:37.20021537 +0000 UTC m=+625.925837138" Nov 02 13:54:37.418331 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:37.418291 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:38.461429 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:38.461391 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:38.868242 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:38.868158 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:39.063879 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:39.063836 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:54:39.430420 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:39.430328 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:40.204404 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:40.204362 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:40.503349 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:40.503237 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:41.110912 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:41.110877 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 13:54:41.458784 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:41.458755 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:41.646818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:41.646756 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:41.827314 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:41.827245 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:42.083518 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:42.083438 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 13:54:42.083670 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:42.083520 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 13:54:42.516550 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:42.516513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:43.111317 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:43.111284 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:43.123411 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:43.123385 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 13:54:43.375057 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:43.374482 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:43.610261 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:43.610227 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:44.310912 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.310879 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj"] Nov 02 13:54:44.316197 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.316174 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.361010 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.354023 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj"] Nov 02 13:54:44.364604 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.364580 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"simulator-w0-1a562c-kserve-self-signed-certs\"" Nov 02 13:54:44.364852 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.364839 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"default-dockercfg-clf8h\"" Nov 02 13:54:44.389924 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.389896 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"kube-root-ca.crt\"" Nov 02 13:54:44.390075 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.389938 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"openshift-service-ca.crt\"" Nov 02 13:54:44.429159 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.429128 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:44.462945 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.462906 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-tmp-dir\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.462945 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.462944 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-home\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.463197 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.462968 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab68e373-a771-4e18-b873-d46cba1c4232-tls-certs\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.463197 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.463146 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-model-cache\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.463270 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.463209 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-dshm\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.463270 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.463244 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8hb\" (UniqueName: \"kubernetes.io/projected/ab68e373-a771-4e18-b873-d46cba1c4232-kube-api-access-zg8hb\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.564522 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-model-cache\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.564615 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-dshm\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.564641 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8hb\" (UniqueName: \"kubernetes.io/projected/ab68e373-a771-4e18-b873-d46cba1c4232-kube-api-access-zg8hb\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.564701 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-tmp-dir\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.564723 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-home\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.564746 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab68e373-a771-4e18-b873-d46cba1c4232-tls-certs\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.565895 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-model-cache\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.566440 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-tmp-dir\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.568685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.566660 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-home\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.573937 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.573903 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-dshm\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.577523 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.575802 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab68e373-a771-4e18-b873-d46cba1c4232-tls-certs\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.593526 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.592115 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8hb\" (UniqueName: \"kubernetes.io/projected/ab68e373-a771-4e18-b873-d46cba1c4232-kube-api-access-zg8hb\") pod \"simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.665032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.664963 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:44.992293 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:44.992264 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj"] Nov 02 13:54:44.994004 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:54:44.993956 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab68e373_a771_4e18_b873_d46cba1c4232.slice/crio-e324376604d7cb87473c9a6bd64117e6cf5a537106734b629eda2aabbb835688 WatchSource:0}: Error finding container e324376604d7cb87473c9a6bd64117e6cf5a537106734b629eda2aabbb835688: Status 404 returned error can't find the container with id e324376604d7cb87473c9a6bd64117e6cf5a537106734b629eda2aabbb835688 Nov 02 13:54:45.180597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:45.180556 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" event={"ID":"ab68e373-a771-4e18-b873-d46cba1c4232","Type":"ContainerStarted","Data":"496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b"} Nov 02 13:54:45.180597 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:45.180598 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" event={"ID":"ab68e373-a771-4e18-b873-d46cba1c4232","Type":"ContainerStarted","Data":"e324376604d7cb87473c9a6bd64117e6cf5a537106734b629eda2aabbb835688"} Nov 02 13:54:45.493307 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:45.493275 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:45.665931 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:45.665899 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:47.223371 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:47.223328 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:47.390823 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:47.390789 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:47.439048 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:47.436943 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podStartSLOduration=3.436925321 podStartE2EDuration="3.436925321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:45.228665283 +0000 UTC m=+633.954287054" watchObservedRunningTime="2026-11-02 13:54:47.436925321 +0000 UTC m=+636.162547091" Nov 02 13:54:47.448172 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:47.448142 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:54:47.599875 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:47.599772 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:48.119685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:48.119655 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 13:54:49.185996 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:49.185849 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:49.204161 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:49.204130 5591 generic.go:356] "Generic (PLEG): container finished" podID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerID="4099ca9e33f9c33e61dcc098d110a5494783071df09f4a113933c8aeeb589219" exitCode=0 Nov 02 13:54:49.204312 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:49.204202 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" event={"ID":"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57","Type":"ContainerDied","Data":"4099ca9e33f9c33e61dcc098d110a5494783071df09f4a113933c8aeeb589219"} Nov 02 13:54:49.204599 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:49.204585 5591 scope.go:122] "RemoveContainer" containerID="4099ca9e33f9c33e61dcc098d110a5494783071df09f4a113933c8aeeb589219" Nov 02 13:54:49.205844 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:49.205819 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:49.205906 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:49.205885 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:49.205942 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:49.205916 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:50.083890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:50.083843 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:50.215173 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:50.215133 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" event={"ID":"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57","Type":"ContainerStarted","Data":"6e7c7485172de18089267f9e00153915b3527cba3449021dbbfa794fd50d21b1"} Nov 02 13:54:50.397070 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:50.396295 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:50.607271 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:50.607241 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:50.999139 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:50.999105 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:50.999410 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:50.999294 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 13:54:50.999410 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:50.999340 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 13:54:51.077427 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:51.077390 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:51.285787 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:51.285694 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:51.488923 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:51.488891 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:51.999477 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:51.999430 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 13:54:51.999668 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:51.999488 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 13:54:52.999356 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:52.999271 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 13:54:52.999356 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:52.999321 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 13:54:53.129282 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:53.129248 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:53.999083 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:53.998971 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 13:54:53.999083 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:53.999057 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 13:54:54.665287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:54.665247 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:54.680020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:54.679972 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:54.766647 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:54.766608 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:54.999444 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:54.999354 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 13:54:54.999444 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:54.999415 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 13:54:55.276295 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:55.276207 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 13:54:55.988347 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:55.988307 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:56.003778 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:56.003748 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:56.004122 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:56.004097 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:56.004742 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:56.004724 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 13:54:56.466892 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:56.466854 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:57.067111 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:57.067077 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:57.668574 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:57.668542 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:57.897234 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:57.897199 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:58.280932 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:58.280902 5591 generic.go:356] "Generic (PLEG): container finished" podID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerID="fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03" exitCode=0 Nov 02 13:54:58.281394 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:58.280993 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" event={"ID":"8e7d3fe4-0505-4a74-a801-90cd65ea1a41","Type":"ContainerDied","Data":"fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03"} Nov 02 13:54:58.281394 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:58.281329 5591 scope.go:122] "RemoveContainer" containerID="fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03" Nov 02 13:54:58.282523 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:58.282499 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:58.282601 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:58.282563 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:58.282601 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:58.282596 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:54:58.749651 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:58.749620 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:59.059914 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:59.059830 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:54:59.288529 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:54:59.288495 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" event={"ID":"8e7d3fe4-0505-4a74-a801-90cd65ea1a41","Type":"ContainerStarted","Data":"4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a"} Nov 02 13:55:00.056150 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:00.056108 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:55:00.075377 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:00.075351 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:55:00.076685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:00.076660 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 13:55:00.076794 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:00.076702 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 13:55:00.618111 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:00.618072 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:01.075679 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:01.075646 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 13:55:01.075883 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:01.075697 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 13:55:01.081236 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:55:01.081203 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:55:02.075406 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:02.075365 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 13:55:02.075949 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:02.075435 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 13:55:02.664043 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:02.664005 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:03.075311 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:03.075225 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 13:55:03.075311 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:03.075280 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 13:55:03.557518 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:03.557480 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:04.076066 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:04.076025 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 13:55:04.076506 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:04.076093 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 13:55:04.120190 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:04.120150 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:04.406919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:04.406885 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:05.079782 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.079747 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:55:05.080917 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.080894 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 13:55:05.752932 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.752899 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk"] Nov 02 13:55:05.756441 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.756410 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.756961 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.756934 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-model-cache\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.757107 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.756975 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-dshm\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.757107 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.757102 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc692aee-a900-41cb-82ef-911d588161f3-tls-certs\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.757223 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.757194 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-home\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.757278 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.757259 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87t5t\" (UniqueName: \"kubernetes.io/projected/bc692aee-a900-41cb-82ef-911d588161f3-kube-api-access-87t5t\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.757331 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.757290 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-tmp-dir\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.778794 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.778758 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"openshift-service-ca.crt\"" Nov 02 13:55:05.780543 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.780518 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk"] Nov 02 13:55:05.791592 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.791568 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"simulator-w3-f30173-kserve-self-signed-certs\"" Nov 02 13:55:05.792108 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.792081 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"default-dockercfg-s5wk6\"" Nov 02 13:55:05.858172 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858137 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87t5t\" (UniqueName: \"kubernetes.io/projected/bc692aee-a900-41cb-82ef-911d588161f3-kube-api-access-87t5t\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.858380 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858179 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-tmp-dir\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.858380 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858209 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-model-cache\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.858380 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858234 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-dshm\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.858380 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858288 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc692aee-a900-41cb-82ef-911d588161f3-tls-certs\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.858380 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858370 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-home\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.858678 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858656 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-tmp-dir\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.858744 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858682 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-model-cache\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.858744 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.858724 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-home\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.860668 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.860638 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-dshm\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.860899 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.860884 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc692aee-a900-41cb-82ef-911d588161f3-tls-certs\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:05.874489 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.874463 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:05.879023 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.879004 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"kube-root-ca.crt\"" Nov 02 13:55:05.882499 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:05.882477 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87t5t\" (UniqueName: \"kubernetes.io/projected/bc692aee-a900-41cb-82ef-911d588161f3-kube-api-access-87t5t\") pod \"simulator-w3-f30173-kserve-5c54fbb6c6-96cpk\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:06.068003 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:06.067863 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:06.078452 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:06.078374 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:06.416383 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:06.416192 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk"] Nov 02 13:55:07.338351 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:07.338319 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" event={"ID":"bc692aee-a900-41cb-82ef-911d588161f3","Type":"ContainerStarted","Data":"f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8"} Nov 02 13:55:07.338523 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:07.338359 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" event={"ID":"bc692aee-a900-41cb-82ef-911d588161f3","Type":"ContainerStarted","Data":"9ab0f95536b5bd79c7a7d20660aebacfe886bbf87970ff6141c84ad34aa25952"} Nov 02 13:55:07.382657 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:07.382600 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podStartSLOduration=2.382588657 podStartE2EDuration="2.382588657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:07.381303901 +0000 UTC m=+656.106925671" watchObservedRunningTime="2026-11-02 13:55:07.382588657 +0000 UTC m=+656.108210428" Nov 02 13:55:07.881323 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:07.881290 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:08.068705 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:08.068669 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:09.144183 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:09.144140 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:55:09.473221 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:09.473135 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:10.979792 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:10.979752 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:11.282626 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:11.282502 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:13.022553 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:13.022514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:14.835402 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:14.835364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:15.536422 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.536387 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr"] Nov 02 13:55:15.540681 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.540648 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.552328 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.552295 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft54l\" (UniqueName: \"kubernetes.io/projected/79dc1a59-b95e-4f19-b397-c39eef342da2-kube-api-access-ft54l\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.552456 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.552347 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-home\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.552456 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.552431 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-tmp-dir\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.552532 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.552456 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79dc1a59-b95e-4f19-b397-c39eef342da2-tls-certs\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.552532 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.552479 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-dshm\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.552595 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.552571 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-model-cache\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.562283 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.562258 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"simulator-w1-92f249-kserve-self-signed-certs\"" Nov 02 13:55:15.575416 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.575388 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"default-dockercfg-b9x6d\"" Nov 02 13:55:15.577009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.576974 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr"] Nov 02 13:55:15.578400 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.578368 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"openshift-service-ca.crt\"" Nov 02 13:55:15.578523 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.578414 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"kube-root-ca.crt\"" Nov 02 13:55:15.653087 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653050 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-dshm\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.653287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653103 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-model-cache\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.653287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653251 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft54l\" (UniqueName: \"kubernetes.io/projected/79dc1a59-b95e-4f19-b397-c39eef342da2-kube-api-access-ft54l\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.653414 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653290 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-home\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.653414 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653338 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-tmp-dir\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.653414 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653365 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79dc1a59-b95e-4f19-b397-c39eef342da2-tls-certs\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.653564 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653543 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-model-cache\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.653748 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653729 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-tmp-dir\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.653822 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.653774 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-home\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.655524 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.655505 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-dshm\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.656018 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.655999 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79dc1a59-b95e-4f19-b397-c39eef342da2-tls-certs\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.669077 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.669038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:15.681377 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.681347 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft54l\" (UniqueName: \"kubernetes.io/projected/79dc1a59-b95e-4f19-b397-c39eef342da2-kube-api-access-ft54l\") pod \"simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:15.853168 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:15.853085 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:16.069359 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.069319 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:16.082043 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.082012 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:16.153076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.153049 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr"] Nov 02 13:55:16.154660 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:55:16.154628 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79dc1a59_b95e_4f19_b397_c39eef342da2.slice/crio-75eecad4e9c5d52d2838ca788b9dbc6363c98777f8b3427babc99b3506f3e157 WatchSource:0}: Error finding container 75eecad4e9c5d52d2838ca788b9dbc6363c98777f8b3427babc99b3506f3e157: Status 404 returned error can't find the container with id 75eecad4e9c5d52d2838ca788b9dbc6363c98777f8b3427babc99b3506f3e157 Nov 02 13:55:16.389362 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.389320 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" event={"ID":"79dc1a59-b95e-4f19-b397-c39eef342da2","Type":"ContainerStarted","Data":"822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab"} Nov 02 13:55:16.389562 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.389371 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" event={"ID":"79dc1a59-b95e-4f19-b397-c39eef342da2","Type":"ContainerStarted","Data":"75eecad4e9c5d52d2838ca788b9dbc6363c98777f8b3427babc99b3506f3e157"} Nov 02 13:55:16.400708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.400685 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:55:16.435230 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.435133 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podStartSLOduration=1.435121072 podStartE2EDuration="1.435121072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:16.433776072 +0000 UTC m=+665.159397841" watchObservedRunningTime="2026-11-02 13:55:16.435121072 +0000 UTC m=+665.160742842" Nov 02 13:55:16.573941 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.573907 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:16.853672 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:16.853590 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:18.292193 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:18.292151 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:19.270072 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:19.270037 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:55:19.870816 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:19.870781 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:20.589732 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:20.589697 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:21.560673 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:21.560637 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:22.689738 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:22.689701 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:23.435405 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:23.435373 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:24.312603 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.312565 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:24.632504 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.632477 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4"] Nov 02 13:55:24.635178 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.635162 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.665241 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.665205 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"default-dockercfg-9z88n\"" Nov 02 13:55:24.665241 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.665218 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"kube-root-ca.crt\"" Nov 02 13:55:24.665635 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.665232 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"premium-w2-0969b1-kserve-self-signed-certs\"" Nov 02 13:55:24.669525 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.669497 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4"] Nov 02 13:55:24.722255 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.722220 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:24.748413 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.748382 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-dshm\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.748607 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.748453 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xxr\" (UniqueName: \"kubernetes.io/projected/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-kube-api-access-98xxr\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.748607 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.748516 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tmp-dir\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.748607 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.748584 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tls-certs\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.748776 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.748611 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-home\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.748776 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.748650 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-model-cache\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.758921 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.758899 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"openshift-service-ca.crt\"" Nov 02 13:55:24.849961 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.849927 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-dshm\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.850180 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.850003 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xxr\" (UniqueName: \"kubernetes.io/projected/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-kube-api-access-98xxr\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.850180 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.850063 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tmp-dir\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.850180 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.850094 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tls-certs\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.850180 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.850118 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-home\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.850180 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.850161 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-model-cache\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.850517 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.850494 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tmp-dir\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.850580 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.850536 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-home\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.850580 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.850554 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-model-cache\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.852276 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.852256 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-dshm\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.852554 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.852538 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tls-certs\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.878054 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.878028 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xxr\" (UniqueName: \"kubernetes.io/projected/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-kube-api-access-98xxr\") pod \"premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:24.945905 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:24.945816 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:25.182083 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:25.182051 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:25.246220 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:25.246192 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4"] Nov 02 13:55:25.248365 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:55:25.248336 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01723d00_e2c0_4ab1_bbee_ff8a7d6ac179.slice/crio-bab878d164fb18126c732c96ce20dc869b3806c0fe5316bdbf2dd778fea5c657 WatchSource:0}: Error finding container bab878d164fb18126c732c96ce20dc869b3806c0fe5316bdbf2dd778fea5c657: Status 404 returned error can't find the container with id bab878d164fb18126c732c96ce20dc869b3806c0fe5316bdbf2dd778fea5c657 Nov 02 13:55:25.440398 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:25.440363 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" event={"ID":"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179","Type":"ContainerStarted","Data":"22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca"} Nov 02 13:55:25.440779 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:25.440408 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" event={"ID":"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179","Type":"ContainerStarted","Data":"bab878d164fb18126c732c96ce20dc869b3806c0fe5316bdbf2dd778fea5c657"} Nov 02 13:55:25.484024 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:25.483956 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podStartSLOduration=1.48394411 podStartE2EDuration="1.48394411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:25.483370948 +0000 UTC m=+674.208992712" watchObservedRunningTime="2026-11-02 13:55:25.48394411 +0000 UTC m=+674.209565880" Nov 02 13:55:25.854173 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:25.854129 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:25.869668 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:25.869638 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:25.946098 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:25.946046 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:26.456179 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:26.456150 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:55:26.968010 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:26.966380 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:28.478831 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:28.478798 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:55:28.669611 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:28.669571 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:30.464811 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:30.464662 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:31.072924 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:55:31.072887 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:55:32.149974 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:32.149927 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:33.760890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:33.760851 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:34.946079 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:34.946034 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:34.959162 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:34.959130 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:35.479649 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:35.479613 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:35.505505 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:35.505471 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 13:55:37.170603 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:37.170561 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:38.762818 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:38.762763 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:39.385508 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:39.385465 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:40.363335 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:40.363302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:41.957934 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:41.957896 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:43.551249 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:43.551198 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:43.933256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:43.933215 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:55:45.366026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:45.365959 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:46.392504 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.392470 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc"] Nov 02 13:55:46.397631 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.397602 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.416314 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.416282 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc"] Nov 02 13:55:46.426593 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.426568 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"premium-w3-f30173-kserve-self-signed-certs\"" Nov 02 13:55:46.485280 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.485245 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-home\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.485462 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.485289 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lcx\" (UniqueName: \"kubernetes.io/projected/f4d6146c-41ab-41c7-a5f5-222f67358b69-kube-api-access-84lcx\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.485462 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.485377 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-tmp-dir\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.485561 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.485544 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d6146c-41ab-41c7-a5f5-222f67358b69-tls-certs\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.485642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.485623 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-dshm\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.485711 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.485673 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-model-cache\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.519117 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.519079 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:46.586711 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.586667 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lcx\" (UniqueName: \"kubernetes.io/projected/f4d6146c-41ab-41c7-a5f5-222f67358b69-kube-api-access-84lcx\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.587858 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.587825 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-tmp-dir\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.588203 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.588183 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d6146c-41ab-41c7-a5f5-222f67358b69-tls-certs\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.588385 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.588366 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-dshm\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.589324 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.589295 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-model-cache\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.589448 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.589347 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-home\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.589923 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.589792 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-home\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.590877 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.590846 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-tmp-dir\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.592624 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.592597 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-model-cache\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.612760 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.607905 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d6146c-41ab-41c7-a5f5-222f67358b69-tls-certs\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.613599 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.613535 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lcx\" (UniqueName: \"kubernetes.io/projected/f4d6146c-41ab-41c7-a5f5-222f67358b69-kube-api-access-84lcx\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.614148 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.614118 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-dshm\") pod \"premium-w3-f30173-kserve-6b5d65cbb5-wqlqc\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.722415 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.722338 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:46.962008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:46.960560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:47.021242 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:47.021219 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc"] Nov 02 13:55:47.022539 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:55:47.022511 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4d6146c_41ab_41c7_a5f5_222f67358b69.slice/crio-42545f2054cdef83310c46e548b35a321b4ff377a256a061ea07fbb8159cca39 WatchSource:0}: Error finding container 42545f2054cdef83310c46e548b35a321b4ff377a256a061ea07fbb8159cca39: Status 404 returned error can't find the container with id 42545f2054cdef83310c46e548b35a321b4ff377a256a061ea07fbb8159cca39 Nov 02 13:55:47.558999 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:47.558952 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" event={"ID":"f4d6146c-41ab-41c7-a5f5-222f67358b69","Type":"ContainerStarted","Data":"6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523"} Nov 02 13:55:47.559498 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:47.559007 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" event={"ID":"f4d6146c-41ab-41c7-a5f5-222f67358b69","Type":"ContainerStarted","Data":"42545f2054cdef83310c46e548b35a321b4ff377a256a061ea07fbb8159cca39"} Nov 02 13:55:47.603411 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:47.603361 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podStartSLOduration=1.603348026 podStartE2EDuration="1.603348026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:47.602047251 +0000 UTC m=+696.327669021" watchObservedRunningTime="2026-11-02 13:55:47.603348026 +0000 UTC m=+696.328969808" Nov 02 13:55:47.725774 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:47.725704 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:48.949407 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:48.949370 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:49.965283 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:49.965248 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:50.432465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:50.432414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:52.279246 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:52.279209 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:54.062579 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:54.062521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:55.765157 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:55.765110 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:56.723234 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:56.723193 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:56.736265 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:56.736232 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:57.479599 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:57.479565 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:57.620463 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:57.620437 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 13:55:59.194070 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:55:59.194031 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:00.755630 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:00.755595 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:56:00.853256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:00.853224 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:01.073330 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:56:01.073237 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:56:02.703727 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:02.703695 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:04.357604 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:04.355963 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:05.953924 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:05.953799 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:07.839140 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:07.839105 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:09.460953 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:09.460911 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:11.353501 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:11.353451 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:13.161255 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:13.161194 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:14.852018 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:14.851307 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:16.482915 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:16.482865 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:18.056534 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:18.056495 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:19.848012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:19.847936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:20.677824 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:20.677789 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:21.566812 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:21.566740 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:21.830931 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:21.830849 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:56:23.344786 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:23.344588 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:24.475356 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.475324 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s"] Nov 02 13:56:24.480412 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.480391 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.514749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.514714 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1a562c-openshift-default-dockercfg-smz6x\"" Nov 02 13:56:24.516486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.516462 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s"] Nov 02 13:56:24.616085 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616048 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.616257 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616103 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.616257 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616192 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc55x\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-kube-api-access-fc55x\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.616257 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616227 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.616419 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616335 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.616419 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616365 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.616521 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616435 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.616521 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616465 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.616521 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.616501 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2465170c-9935-4a55-a00d-7dcaeeb78338-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.717975 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.717941 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2465170c-9935-4a55-a00d-7dcaeeb78338-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718160 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718054 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718160 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718084 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718160 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718102 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc55x\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-kube-api-access-fc55x\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718160 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718128 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718387 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718179 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718387 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718205 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718496 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718407 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718496 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718458 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718756 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718707 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718756 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718750 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718770 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718797 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2465170c-9935-4a55-a00d-7dcaeeb78338-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.718910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.718808 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.720648 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.720629 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.721114 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.721096 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.742904 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.742854 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.743673 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.743657 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc55x\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-kube-api-access-fc55x\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:24.793483 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:24.793448 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:25.062269 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.062190 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:25.121549 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.121520 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s"] Nov 02 13:56:25.123399 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:56:25.123374 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2465170c_9935_4a55_a00d_7dcaeeb78338.slice/crio-d5c9b2e9b75bb3c5d8abd88cb89890a7496938dcc3d1d4eb77f0df8566527eda WatchSource:0}: Error finding container d5c9b2e9b75bb3c5d8abd88cb89890a7496938dcc3d1d4eb77f0df8566527eda: Status 404 returned error can't find the container with id d5c9b2e9b75bb3c5d8abd88cb89890a7496938dcc3d1d4eb77f0df8566527eda Nov 02 13:56:25.126112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.126064 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:56:25.126257 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.126233 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:56:25.126341 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.126321 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:56:25.757157 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.757119 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" event={"ID":"2465170c-9935-4a55-a00d-7dcaeeb78338","Type":"ContainerStarted","Data":"f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66"} Nov 02 13:56:25.757678 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.757178 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" event={"ID":"2465170c-9935-4a55-a00d-7dcaeeb78338","Type":"ContainerStarted","Data":"d5c9b2e9b75bb3c5d8abd88cb89890a7496938dcc3d1d4eb77f0df8566527eda"} Nov 02 13:56:25.793574 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.793547 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:25.804767 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:25.804718 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podStartSLOduration=1.8047044319999999 podStartE2EDuration="1.804704432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:25.802708377 +0000 UTC m=+734.528330147" watchObservedRunningTime="2026-11-02 13:56:25.804704432 +0000 UTC m=+734.530326202" Nov 02 13:56:26.763489 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:26.761855 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:26.794866 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:26.794834 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 13:56:26.795081 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:26.794923 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 13:56:27.799410 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:27.799379 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 13:56:27.800186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:27.800150 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 13:56:28.505975 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:28.505936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:28.794634 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:28.794551 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 13:56:28.794634 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:28.794615 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 13:56:29.794152 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:29.794116 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 13:56:29.794557 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:29.794186 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 13:56:30.198730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:30.198698 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:30.794845 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:30.794807 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 13:56:30.795283 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:30.794874 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 13:56:31.081543 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:56:31.081452 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:56:31.794175 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:31.794137 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 13:56:31.794342 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:31.794225 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 13:56:32.076087 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:32.075490 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:32.798496 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:32.798467 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:32.798758 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:32.798735 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:32.799527 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:32.799509 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 13:56:33.663111 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:33.663048 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:35.389191 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:35.389153 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:37.210018 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:37.209960 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:38.876546 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:38.876511 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:40.460334 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:40.460298 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:42.065657 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:42.065600 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:43.750167 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:43.750111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:45.467893 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:45.467853 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:47.177669 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:47.177627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:47.736292 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:47.736259 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:47.922895 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:47.922846 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:49.053008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.050916 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:49.274254 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.274222 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:49.484992 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.484952 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-597956f59c-4q72h"] Nov 02 13:56:49.488789 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.488771 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.523293 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.523263 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4ab4173f18e5\"" Nov 02 13:56:49.523520 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.523266 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-994ede8836ba\"" Nov 02 13:56:49.523520 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.523321 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 13:56:49.523680 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.523359 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 13:56:49.523680 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.523362 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 13:56:49.523680 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.523399 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 13:56:49.524939 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.524910 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lzlpd\"" Nov 02 13:56:49.530047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.530022 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-597956f59c-4q72h"] Nov 02 13:56:49.567665 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.567616 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbr5\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-kube-api-access-rpbr5\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.569279 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.569247 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-provider-credentials\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.569642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.569621 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-plugins-config-volume\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.569834 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.569806 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-routing-overlay-volume\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.671480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.671434 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-plugins-config-volume\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.671679 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.671600 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-routing-overlay-volume\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.671679 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.671651 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbr5\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-kube-api-access-rpbr5\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.671791 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.671681 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-provider-credentials\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.672489 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.672454 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-routing-overlay-volume\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.672678 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.672482 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-plugins-config-volume\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.674707 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.674676 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-provider-credentials\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.701538 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.701497 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbr5\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-kube-api-access-rpbr5\") pod \"payload-processing-external-model-597956f59c-4q72h\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:49.802793 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:49.802698 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:50.115702 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:50.115659 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-597956f59c-4q72h"] Nov 02 13:56:50.117449 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:56:50.117412 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc628596f_5762_47be_9f5a_15cb3c2cf60d.slice/crio-db150b09de6b8bd244524ec4cfe0f04e5806cfcce31c96e65a8a67592c9cb24d WatchSource:0}: Error finding container db150b09de6b8bd244524ec4cfe0f04e5806cfcce31c96e65a8a67592c9cb24d: Status 404 returned error can't find the container with id db150b09de6b8bd244524ec4cfe0f04e5806cfcce31c96e65a8a67592c9cb24d Nov 02 13:56:50.772008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:50.768810 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:50.921469 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:50.921429 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" event={"ID":"c628596f-5762-47be-9f5a-15cb3c2cf60d","Type":"ContainerStarted","Data":"378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b"} Nov 02 13:56:50.921644 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:50.921489 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" event={"ID":"c628596f-5762-47be-9f5a-15cb3c2cf60d","Type":"ContainerStarted","Data":"db150b09de6b8bd244524ec4cfe0f04e5806cfcce31c96e65a8a67592c9cb24d"} Nov 02 13:56:50.921644 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:50.921530 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:56:50.983007 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:50.981951 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" podStartSLOduration=1.981929654 podStartE2EDuration="1.981929654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:50.979585661 +0000 UTC m=+759.705207431" watchObservedRunningTime="2026-11-02 13:56:50.981929654 +0000 UTC m=+759.707551426" Nov 02 13:56:52.467632 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:52.467593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:53.443725 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:53.443692 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:56:54.158012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:54.157792 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:55.964423 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:55.964385 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:57.580377 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:57.580197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:59.509046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:56:59.509008 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:01.074071 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:57:01.074036 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:57:01.308155 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:01.308116 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:01.808131 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:01.808102 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:01.930453 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:01.930425 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:57:02.006195 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:02.006162 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:03.210699 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.210664 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff6674646-szp4c"] Nov 02 13:57:03.215125 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.215101 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.233267 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.233239 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3eee531c3de6\"" Nov 02 13:57:03.233406 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.233323 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-892a32dda57a\"" Nov 02 13:57:03.235298 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.235272 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff6674646-szp4c"] Nov 02 13:57:03.284088 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.284055 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wgx\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-kube-api-access-x7wgx\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.284267 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.284186 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-routing-overlay-volume\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.284267 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.284236 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-provider-credentials\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.284373 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.284268 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-plugins-config-volume\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.348019 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.339478 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:03.385865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.385834 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-routing-overlay-volume\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.385865 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.385871 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-provider-credentials\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.386157 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.386066 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-plugins-config-volume\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.386205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.386154 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wgx\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-kube-api-access-x7wgx\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.386499 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.386477 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-routing-overlay-volume\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.386912 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.386888 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-plugins-config-volume\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.388927 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.388903 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-provider-credentials\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.411461 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.411425 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wgx\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-kube-api-access-x7wgx\") pod \"payload-processing-external-model-7ff6674646-szp4c\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.529152 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.529075 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:03.835005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:03.834948 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff6674646-szp4c"] Nov 02 13:57:03.836572 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:57:03.836548 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e632f32_7ef7_4fbc_866e_3c6d8ad7b0a6.slice/crio-2b949b410bfeef54758db37d4ecd2db5b421afa8f947448bbe7269d28d1deca8 WatchSource:0}: Error finding container 2b949b410bfeef54758db37d4ecd2db5b421afa8f947448bbe7269d28d1deca8: Status 404 returned error can't find the container with id 2b949b410bfeef54758db37d4ecd2db5b421afa8f947448bbe7269d28d1deca8 Nov 02 13:57:04.025563 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.025523 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" event={"ID":"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6","Type":"ContainerStarted","Data":"be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee"} Nov 02 13:57:04.025762 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.025576 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" event={"ID":"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6","Type":"ContainerStarted","Data":"2b949b410bfeef54758db37d4ecd2db5b421afa8f947448bbe7269d28d1deca8"} Nov 02 13:57:04.025762 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.025641 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:04.069475 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.069420 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" podStartSLOduration=1.069405678 podStartE2EDuration="1.069405678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:04.068521427 +0000 UTC m=+772.794143196" watchObservedRunningTime="2026-11-02 13:57:04.069405678 +0000 UTC m=+772.795027448" Nov 02 13:57:04.544322 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.544288 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24"] Nov 02 13:57:04.548215 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.548196 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.566839 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.566814 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-92f249-openshift-default-dockercfg-p2t2n\"" Nov 02 13:57:04.581807 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.581726 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24"] Nov 02 13:57:04.598578 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.598542 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c18551-b05a-4964-9964-dd5d7a79e943-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.598737 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.598616 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.598737 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.598699 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.598857 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.598753 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.598857 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.598841 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfph\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-kube-api-access-rnfph\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.598936 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.598903 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.599004 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.598937 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c18551-b05a-4964-9964-dd5d7a79e943-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.599099 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.599079 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.599145 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.599114 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700054 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700005 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700072 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfph\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-kube-api-access-rnfph\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700093 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700115 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c18551-b05a-4964-9964-dd5d7a79e943-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700180 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700256 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700208 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700306 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c18551-b05a-4964-9964-dd5d7a79e943-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700371 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700435 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700495 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700703 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700820 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700797 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700815 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.700882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.700839 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c18551-b05a-4964-9964-dd5d7a79e943-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.702470 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.702445 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.702792 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.702774 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c18551-b05a-4964-9964-dd5d7a79e943-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.725586 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.725561 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.725704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.725616 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfph\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-kube-api-access-rnfph\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:04.861769 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:04.861689 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:05.076886 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:05.076853 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:05.188103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:05.188078 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24"] Nov 02 13:57:05.189908 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:57:05.189885 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c18551_b05a_4964_9964_dd5d7a79e943.slice/crio-22fe963ac7efbb0d2c74d941fe4bbc776547ec4ab6ad0191c8dc9c44195478f7 WatchSource:0}: Error finding container 22fe963ac7efbb0d2c74d941fe4bbc776547ec4ab6ad0191c8dc9c44195478f7: Status 404 returned error can't find the container with id 22fe963ac7efbb0d2c74d941fe4bbc776547ec4ab6ad0191c8dc9c44195478f7 Nov 02 13:57:05.192255 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:05.192226 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:57:05.192320 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:05.192289 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:57:05.192409 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:05.192317 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:57:06.059810 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:06.059776 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" event={"ID":"14c18551-b05a-4964-9964-dd5d7a79e943","Type":"ContainerStarted","Data":"4c10841914bc015c30005f36f9ad33e58c234423389d1498718c157aacbdb20e"} Nov 02 13:57:06.060234 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:06.059833 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" event={"ID":"14c18551-b05a-4964-9964-dd5d7a79e943","Type":"ContainerStarted","Data":"22fe963ac7efbb0d2c74d941fe4bbc776547ec4ab6ad0191c8dc9c44195478f7"} Nov 02 13:57:06.151015 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:06.146382 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podStartSLOduration=2.146346088 podStartE2EDuration="2.146346088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:06.121235691 +0000 UTC m=+774.846857460" watchObservedRunningTime="2026-11-02 13:57:06.146346088 +0000 UTC m=+774.871967858" Nov 02 13:57:06.864369 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:06.864302 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:06.867873 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:06.867836 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 13:57:06.868140 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:06.868105 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 13:57:06.892011 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:06.888939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:07.864796 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:07.864347 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 13:57:07.864796 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:07.864412 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 13:57:08.670004 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:08.669482 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:08.865008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:08.862261 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 13:57:08.865008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:08.862320 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 13:57:09.865011 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:09.863437 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 13:57:09.865011 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:09.863524 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 13:57:10.280506 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:10.280471 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:10.862718 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:10.862681 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 13:57:10.862899 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:10.862741 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 13:57:11.862446 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:11.862412 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 13:57:11.862855 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:11.862467 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 13:57:12.255599 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:12.255560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:12.866098 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:12.866066 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:12.866560 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:12.866371 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:12.867056 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:12.867037 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 13:57:13.867943 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:13.867848 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:15.033863 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:15.033827 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:57:15.111235 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:15.111206 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-597956f59c-4q72h"] Nov 02 13:57:15.557716 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:15.557644 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:17.317563 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:17.317521 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:57:17.498806 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:17.498757 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:18.924188 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:18.924155 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:19.406005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:19.403315 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:20.111576 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.111529 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" podUID="c628596f-5762-47be-9f5a-15cb3c2cf60d" containerName="payload-processing" containerID="cri-o://378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b" gracePeriod=25 Nov 02 13:57:20.359035 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.359005 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:57:20.363052 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.362977 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-provider-credentials\") pod \"c628596f-5762-47be-9f5a-15cb3c2cf60d\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " Nov 02 13:57:20.363177 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.363068 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-plugins-config-volume\") pod \"c628596f-5762-47be-9f5a-15cb3c2cf60d\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " Nov 02 13:57:20.363177 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.363140 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-routing-overlay-volume\") pod \"c628596f-5762-47be-9f5a-15cb3c2cf60d\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " Nov 02 13:57:20.363292 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.363187 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-kube-api-access-rpbr5\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-kube-api-access-rpbr5\") pod \"c628596f-5762-47be-9f5a-15cb3c2cf60d\" (UID: \"c628596f-5762-47be-9f5a-15cb3c2cf60d\") " Nov 02 13:57:20.363530 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.363502 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-plugins-config-volume" pod "c628596f-5762-47be-9f5a-15cb3c2cf60d" (UID: "c628596f-5762-47be-9f5a-15cb3c2cf60d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:57:20.363643 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.363498 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-routing-overlay-volume" pod "c628596f-5762-47be-9f5a-15cb3c2cf60d" (UID: "c628596f-5762-47be-9f5a-15cb3c2cf60d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:57:20.365184 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.365160 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-provider-credentials" pod "c628596f-5762-47be-9f5a-15cb3c2cf60d" (UID: "c628596f-5762-47be-9f5a-15cb3c2cf60d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:57:20.365299 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.365285 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-kube-api-access-rpbr5" pod "c628596f-5762-47be-9f5a-15cb3c2cf60d" (UID: "c628596f-5762-47be-9f5a-15cb3c2cf60d"). InnerVolumeSpecName "kube-api-access-rpbr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:57:20.463864 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.463832 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:57:20.463864 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.463859 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:57:20.463864 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.463871 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c628596f-5762-47be-9f5a-15cb3c2cf60d-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:57:20.464141 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.463884 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbr5\" (UniqueName: \"kubernetes.io/projected/c628596f-5762-47be-9f5a-15cb3c2cf60d-kube-api-access-rpbr5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:57:20.810560 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:20.810520 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:21.189912 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.189468 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:21.213362 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.213329 5591 generic.go:356] "Generic (PLEG): container finished" podID="c628596f-5762-47be-9f5a-15cb3c2cf60d" containerID="378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b" exitCode=0 Nov 02 13:57:21.213540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.213380 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" Nov 02 13:57:21.213540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.213382 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" event={"ID":"c628596f-5762-47be-9f5a-15cb3c2cf60d","Type":"ContainerDied","Data":"378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b"} Nov 02 13:57:21.213540 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.213514 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-597956f59c-4q72h" event={"ID":"c628596f-5762-47be-9f5a-15cb3c2cf60d","Type":"ContainerDied","Data":"db150b09de6b8bd244524ec4cfe0f04e5806cfcce31c96e65a8a67592c9cb24d"} Nov 02 13:57:21.213685 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.213544 5591 scope.go:122] "RemoveContainer" containerID="378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b" Nov 02 13:57:21.224184 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.224166 5591 scope.go:122] "RemoveContainer" containerID="378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b" Nov 02 13:57:21.224448 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:57:21.224429 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b\": container with ID starting with 378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b not found: ID does not exist" containerID="378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b" Nov 02 13:57:21.224511 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.224456 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b"} err="failed to get container status \"378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b\": rpc error: code = NotFound desc = could not find container \"378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b\": container with ID starting with 378d12b24bf8e4ff038302a38ad652a69914fa2f9e8574b0d17f7cc498d22e9b not found: ID does not exist" Nov 02 13:57:21.266754 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.266724 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-597956f59c-4q72h"] Nov 02 13:57:21.279714 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.279686 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-597956f59c-4q72h"] Nov 02 13:57:21.431091 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:21.431058 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c628596f-5762-47be-9f5a-15cb3c2cf60d" path="/var/lib/kubelet/pods/c628596f-5762-47be-9f5a-15cb3c2cf60d/volumes" Nov 02 13:57:22.873202 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:22.873126 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:24.484572 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:24.484511 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:26.143612 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:26.143570 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:27.251144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.251106 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf"] Nov 02 13:57:27.251611 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.251597 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c628596f-5762-47be-9f5a-15cb3c2cf60d" containerName="payload-processing" Nov 02 13:57:27.251660 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.251611 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c628596f-5762-47be-9f5a-15cb3c2cf60d" containerName="payload-processing" Nov 02 13:57:27.251699 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.251690 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c628596f-5762-47be-9f5a-15cb3c2cf60d" containerName="payload-processing" Nov 02 13:57:27.254910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.254894 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.281783 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.281754 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf"] Nov 02 13:57:27.354542 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.354507 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.354749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.354551 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.354749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.354636 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.354749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.354684 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.354749 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.354710 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.354919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.354808 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxbr\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-kube-api-access-xsxbr\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.354919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.354848 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.355028 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.354953 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b357daf9-2941-45c8-bd2f-ac9324282013-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.355028 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.355015 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b357daf9-2941-45c8-bd2f-ac9324282013-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.456707 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.456673 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.456882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.456717 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.456882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.456770 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxbr\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-kube-api-access-xsxbr\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.456882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.456796 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.456882 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.456840 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b357daf9-2941-45c8-bd2f-ac9324282013-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.456897 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b357daf9-2941-45c8-bd2f-ac9324282013-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.456966 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.457125 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457275 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.457178 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457275 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.457219 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457275 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.457227 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457474 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.457453 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457594 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.457519 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.457701 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.457684 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b357daf9-2941-45c8-bd2f-ac9324282013-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.459228 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.459206 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.459438 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.459420 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b357daf9-2941-45c8-bd2f-ac9324282013-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.481120 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.481093 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.481268 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.481250 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxbr\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-kube-api-access-xsxbr\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.568397 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.568317 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:27.890700 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.890674 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf"] Nov 02 13:57:27.892700 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:57:27.892676 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb357daf9_2941_45c8_bd2f_ac9324282013.slice/crio-e483954d24979eb2c2ec09c2c12f8d1a91c03324addda20a4ea7a6e9f8f2debb WatchSource:0}: Error finding container e483954d24979eb2c2ec09c2c12f8d1a91c03324addda20a4ea7a6e9f8f2debb: Status 404 returned error can't find the container with id e483954d24979eb2c2ec09c2c12f8d1a91c03324addda20a4ea7a6e9f8f2debb Nov 02 13:57:27.895034 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.894997 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:57:27.895150 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.895083 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:57:27.895150 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.895127 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:57:27.952110 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:27.950619 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:28.252804 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:28.252718 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" event={"ID":"b357daf9-2941-45c8-bd2f-ac9324282013","Type":"ContainerStarted","Data":"95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c"} Nov 02 13:57:28.252804 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:28.252774 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" event={"ID":"b357daf9-2941-45c8-bd2f-ac9324282013","Type":"ContainerStarted","Data":"e483954d24979eb2c2ec09c2c12f8d1a91c03324addda20a4ea7a6e9f8f2debb"} Nov 02 13:57:28.300468 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:28.300421 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podStartSLOduration=1.3004069249999999 podStartE2EDuration="1.300406925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:28.299206168 +0000 UTC m=+797.024827939" watchObservedRunningTime="2026-11-02 13:57:28.300406925 +0000 UTC m=+797.026028721" Nov 02 13:57:28.569325 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:28.569213 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:28.571349 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:28.571319 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 13:57:28.571533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:28.571377 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 13:57:29.569630 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:29.569588 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 13:57:29.570133 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:29.569652 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 13:57:29.874216 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:29.874123 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:30.569586 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:30.569550 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 13:57:30.569747 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:30.569605 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 13:57:31.081656 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:57:31.081614 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:57:31.555695 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:31.555653 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:31.574458 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:31.574416 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 13:57:31.575177 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:31.575091 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 13:57:32.569497 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:32.569459 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 13:57:32.569772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:32.569508 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 13:57:33.350008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:33.342908 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:33.569663 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:33.569624 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 13:57:33.569870 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:33.569697 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 13:57:34.777922 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:34.777891 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:34.778690 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:34.778665 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:34.779810 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:34.779786 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 13:57:35.015012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:35.014969 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:57:35.075063 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:35.067074 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:37.000006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:36.999688 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:37.704504 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:37.704456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:37.896008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:37.895448 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:39.198254 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:39.198220 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:40.994455 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:40.994398 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:42.876824 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:42.876766 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:44.855793 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:44.855756 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:45.093765 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:45.093730 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:47.129733 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:47.129690 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:49.551330 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:49.551294 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:51.503288 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:51.503242 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:53.352185 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:53.348533 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:55.275849 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:55.274405 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:55.682232 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:55.682198 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff6674646-szp4c"] Nov 02 13:57:56.450558 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:56.450522 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:57:57.217619 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:57.217539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:58.107690 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.107657 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bf89bf644-7466n"] Nov 02 13:57:58.111035 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.111016 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.145738 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.145711 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bf89bf644-7466n"] Nov 02 13:57:58.261910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.261880 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-plugins-config-volume\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.261910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.261913 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-provider-credentials\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.262162 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.261957 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45cnp\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-kube-api-access-45cnp\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.262212 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.262179 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-routing-overlay-volume\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.363800 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.363702 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-routing-overlay-volume\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.363800 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.363784 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-plugins-config-volume\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.364072 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.363810 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-provider-credentials\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.364072 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.363862 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45cnp\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-kube-api-access-45cnp\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.364652 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.364625 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-routing-overlay-volume\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.364691 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.364659 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-plugins-config-volume\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.366293 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.366274 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-provider-credentials\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.391103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.391071 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45cnp\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-kube-api-access-45cnp\") pod \"payload-processing-external-model-6bf89bf644-7466n\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.424308 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.424277 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:58.728143 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:58.728119 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bf89bf644-7466n"] Nov 02 13:57:58.729745 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:57:58.729711 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5576506_ec06_4575_8038_cd2e6fda0c07.slice/crio-b6dda2c1cd7c032f5a517b70578dec8b799926304171e3bfe022a4dc565f4f97 WatchSource:0}: Error finding container b6dda2c1cd7c032f5a517b70578dec8b799926304171e3bfe022a4dc565f4f97: Status 404 returned error can't find the container with id b6dda2c1cd7c032f5a517b70578dec8b799926304171e3bfe022a4dc565f4f97 Nov 02 13:57:59.076840 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:59.076749 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:59.467053 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:59.467008 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" event={"ID":"c5576506-ec06-4575-8038-cd2e6fda0c07","Type":"ContainerStarted","Data":"7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a"} Nov 02 13:57:59.467481 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:59.467087 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:57:59.467481 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:59.467110 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" event={"ID":"c5576506-ec06-4575-8038-cd2e6fda0c07","Type":"ContainerStarted","Data":"b6dda2c1cd7c032f5a517b70578dec8b799926304171e3bfe022a4dc565f4f97"} Nov 02 13:57:59.514574 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:57:59.514511 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" podStartSLOduration=1.5144990790000001 podStartE2EDuration="1.514499079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:59.512632808 +0000 UTC m=+828.238254578" watchObservedRunningTime="2026-11-02 13:57:59.514499079 +0000 UTC m=+828.240120875" Nov 02 13:58:00.673026 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:00.669816 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:00.682746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:00.682707 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" podUID="7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" containerName="payload-processing" containerID="cri-o://be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee" gracePeriod=25 Nov 02 13:58:00.937712 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:00.937649 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:58:01.073817 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:58:01.073782 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:58:01.098853 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.098818 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-routing-overlay-volume\") pod \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " Nov 02 13:58:01.099069 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.098879 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-provider-credentials\") pod \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " Nov 02 13:58:01.099069 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.098953 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-kube-api-access-x7wgx\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-kube-api-access-x7wgx\") pod \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " Nov 02 13:58:01.099069 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.099051 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-plugins-config-volume\") pod \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\" (UID: \"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6\") " Nov 02 13:58:01.099378 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.099346 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-routing-overlay-volume" pod "7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" (UID: "7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:01.099479 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.099429 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-plugins-config-volume" pod "7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" (UID: "7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:01.101076 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.101051 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-kube-api-access-x7wgx" pod "7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" (UID: "7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6"). InnerVolumeSpecName "kube-api-access-x7wgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:01.101173 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.101081 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-provider-credentials" pod "7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" (UID: "7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:01.200968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.200879 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:58:01.200968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.200918 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wgx\" (UniqueName: \"kubernetes.io/projected/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-kube-api-access-x7wgx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:58:01.200968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.200930 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:58:01.200968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.200940 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:58:01.480549 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.480459 5591 generic.go:356] "Generic (PLEG): container finished" podID="7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" containerID="be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee" exitCode=0 Nov 02 13:58:01.480549 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.480503 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" event={"ID":"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6","Type":"ContainerDied","Data":"be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee"} Nov 02 13:58:01.480549 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.480533 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" Nov 02 13:58:01.480802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.480554 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl"] Nov 02 13:58:01.480802 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.480577 5591 scope.go:122] "RemoveContainer" containerID="be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee" Nov 02 13:58:01.481919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.481779 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" containerName="payload-processing" Nov 02 13:58:01.481919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.481799 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" containerName="payload-processing" Nov 02 13:58:01.481919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.481915 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" containerName="payload-processing" Nov 02 13:58:01.485689 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.485659 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ff6674646-szp4c" event={"ID":"7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6","Type":"ContainerDied","Data":"2b949b410bfeef54758db37d4ecd2db5b421afa8f947448bbe7269d28d1deca8"} Nov 02 13:58:01.485851 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.485753 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.498765 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.493958 5591 scope.go:122] "RemoveContainer" containerID="be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee" Nov 02 13:58:01.498765 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:58:01.494349 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee\": container with ID starting with be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee not found: ID does not exist" containerID="be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee" Nov 02 13:58:01.498765 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.494381 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee"} err="failed to get container status \"be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee\": rpc error: code = NotFound desc = could not find container \"be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee\": container with ID starting with be4a8b7a311061fd5cb4dee0c9351890d543b2c89c8402733cafbec29ef20eee not found: ID does not exist" Nov 02 13:58:01.516493 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.505334 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"distinct-w3-f30173-kserve-self-signed-certs\"" Nov 02 13:58:01.519469 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.519422 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl"] Nov 02 13:58:01.605085 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.605044 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-model-cache\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.605269 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.605232 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tls-certs\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.605331 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.605313 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcssr\" (UniqueName: \"kubernetes.io/projected/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-kube-api-access-lcssr\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.605404 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.605386 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tmp-dir\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.605555 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.605529 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-dshm\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.605700 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.605583 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-home\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.625863 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.625837 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff6674646-szp4c"] Nov 02 13:58:01.653590 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.653554 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff6674646-szp4c"] Nov 02 13:58:01.680007 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.670485 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.707458 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-model-cache\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.707640 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tls-certs\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.707725 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcssr\" (UniqueName: \"kubernetes.io/projected/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-kube-api-access-lcssr\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.707768 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tmp-dir\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.707817 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-dshm\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.707845 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-home\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.708463 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-home\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.708706 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-model-cache\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.709730 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.709299 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tmp-dir\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.712829 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.712773 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tls-certs\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.714091 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.714052 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-dshm\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.740193 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.740115 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcssr\" (UniqueName: \"kubernetes.io/projected/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-kube-api-access-lcssr\") pod \"distinct-w3-f30173-kserve-766464cb68-p2bxl\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:01.806877 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:01.806840 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:02.107688 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:02.107665 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl"] Nov 02 13:58:02.109451 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:58:02.109421 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea87f45c_22b3_4eb0_a1f9_47bc7c795155.slice/crio-bd0a4b5e50a74bcd41ea1a3191dbf7a106512a6115c260992439bd63f0903814 WatchSource:0}: Error finding container bd0a4b5e50a74bcd41ea1a3191dbf7a106512a6115c260992439bd63f0903814: Status 404 returned error can't find the container with id bd0a4b5e50a74bcd41ea1a3191dbf7a106512a6115c260992439bd63f0903814 Nov 02 13:58:02.481092 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:02.481052 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:02.488609 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:02.488568 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" event={"ID":"ea87f45c-22b3-4eb0-a1f9-47bc7c795155","Type":"ContainerStarted","Data":"9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1"} Nov 02 13:58:02.488813 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:02.488624 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" event={"ID":"ea87f45c-22b3-4eb0-a1f9-47bc7c795155","Type":"ContainerStarted","Data":"bd0a4b5e50a74bcd41ea1a3191dbf7a106512a6115c260992439bd63f0903814"} Nov 02 13:58:02.533578 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:02.533528 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podStartSLOduration=1.533511471 podStartE2EDuration="1.533511471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:02.532537038 +0000 UTC m=+831.258158809" watchObservedRunningTime="2026-11-02 13:58:02.533511471 +0000 UTC m=+831.259133241" Nov 02 13:58:02.807626 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:02.807554 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:03.430633 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:03.430594 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6" path="/var/lib/kubelet/pods/7e632f32-7ef7-4fbc-866e-3c6d8ad7b0a6/volumes" Nov 02 13:58:04.262008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:04.258852 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:04.772542 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:04.772501 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:06.519009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:06.518953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:08.174970 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:08.174931 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:09.757132 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:09.752108 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:09.927165 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:09.927133 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:10.406913 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.406879 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz"] Nov 02 13:58:10.410943 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.410920 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.427708 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.427675 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz"] Nov 02 13:58:10.437527 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.437500 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"distinct-w1-92f249-kserve-self-signed-certs\"" Nov 02 13:58:10.476036 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.476010 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:58:10.496177 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.496142 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-model-cache\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.496343 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.496203 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tls-certs\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.496343 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.496322 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tmp-dir\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.496421 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.496373 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-home\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.496498 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.496478 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw589\" (UniqueName: \"kubernetes.io/projected/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-kube-api-access-xw589\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.496542 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.496527 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-dshm\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.506338 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.506312 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:10.597460 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.597428 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-model-cache\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.597676 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.597503 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tls-certs\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.597676 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.597579 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tmp-dir\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.597676 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.597626 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-home\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.597946 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.597675 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw589\" (UniqueName: \"kubernetes.io/projected/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-kube-api-access-xw589\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.598031 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.598011 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-dshm\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.598085 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.598062 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tmp-dir\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.598147 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.598112 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-model-cache\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.598200 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.598161 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-home\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.599976 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.599950 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-dshm\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.600096 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.600001 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tls-certs\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.622417 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.622387 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw589\" (UniqueName: \"kubernetes.io/projected/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-kube-api-access-xw589\") pod \"distinct-w1-92f249-kserve-7cccbccf96-vclzz\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:10.726920 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:10.726840 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:11.034202 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.034180 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz"] Nov 02 13:58:11.035961 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:58:11.035927 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb347cc67_4e2f_44bd_9d7a_ab4b4774d189.slice/crio-912b5821d2bddf1e2f6e68703ea3aa96a03eb1952f2ceb92a0cb3fc59d0264b4 WatchSource:0}: Error finding container 912b5821d2bddf1e2f6e68703ea3aa96a03eb1952f2ceb92a0cb3fc59d0264b4: Status 404 returned error can't find the container with id 912b5821d2bddf1e2f6e68703ea3aa96a03eb1952f2ceb92a0cb3fc59d0264b4 Nov 02 13:58:11.318876 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.318795 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:58:11.501662 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.500073 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:11.578511 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.578445 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" event={"ID":"b347cc67-4e2f-44bd-9d7a-ab4b4774d189","Type":"ContainerStarted","Data":"47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532"} Nov 02 13:58:11.578511 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.578487 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" event={"ID":"b347cc67-4e2f-44bd-9d7a-ab4b4774d189","Type":"ContainerStarted","Data":"912b5821d2bddf1e2f6e68703ea3aa96a03eb1952f2ceb92a0cb3fc59d0264b4"} Nov 02 13:58:11.621316 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.621265 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podStartSLOduration=1.621252065 podStartE2EDuration="1.621252065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:11.621042188 +0000 UTC m=+840.346663964" watchObservedRunningTime="2026-11-02 13:58:11.621252065 +0000 UTC m=+840.346873836" Nov 02 13:58:11.727901 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.727867 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:11.807934 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.807909 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:11.820576 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:11.820549 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:12.597375 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:12.597340 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 13:58:13.601004 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:13.598256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:15.467281 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:15.467245 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:17.030967 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:17.030919 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:18.949006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:18.948490 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:19.177674 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:19.177637 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:20.576771 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:20.576664 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:20.727622 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:20.727586 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:20.796435 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:20.796397 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:21.003233 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:21.003198 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:21.669139 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:21.669106 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 13:58:22.372599 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:22.369328 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:24.051768 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:24.051712 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:25.660745 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:25.660708 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:27.351254 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:27.351210 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:27.866288 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:27.866252 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6"] Nov 02 13:58:27.870487 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:27.870464 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:27.902107 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:27.902075 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-0969b1-openshift-default-dockercfg-8rlbj\"" Nov 02 13:58:27.910237 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:27.910203 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6"] Nov 02 13:58:28.031130 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031101 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p82m\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-kube-api-access-4p82m\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.031130 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031134 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.031326 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031155 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.031326 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031257 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f128ca-a03d-471b-a951-4d19cddd1785-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.031326 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031281 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.031326 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031297 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f128ca-a03d-471b-a951-4d19cddd1785-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.031544 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031331 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.031544 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031474 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.031642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.031579 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133109 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133077 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f128ca-a03d-471b-a951-4d19cddd1785-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133122 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133143 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f128ca-a03d-471b-a951-4d19cddd1785-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133182 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133209 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133257 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133556 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133312 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p82m\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-kube-api-access-4p82m\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133556 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133340 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133556 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133367 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133721 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133553 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133721 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133679 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133833 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133758 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.133930 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.133834 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.134160 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.134141 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f128ca-a03d-471b-a951-4d19cddd1785-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.135552 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.135529 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.135647 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.135615 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f128ca-a03d-471b-a951-4d19cddd1785-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.157688 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.157663 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.157839 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.157724 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p82m\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-kube-api-access-4p82m\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.183406 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.183380 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:28.501380 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.501350 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6"] Nov 02 13:58:28.504033 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:58:28.504003 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f128ca_a03d_471b_a951_4d19cddd1785.slice/crio-84ed00777f8bbd9254371eddf06d7e66fd2097e940c6d1ee1ac4c6f6d6375f7c WatchSource:0}: Error finding container 84ed00777f8bbd9254371eddf06d7e66fd2097e940c6d1ee1ac4c6f6d6375f7c: Status 404 returned error can't find the container with id 84ed00777f8bbd9254371eddf06d7e66fd2097e940c6d1ee1ac4c6f6d6375f7c Nov 02 13:58:28.506456 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.506426 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:28.506548 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.506489 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:28.506548 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.506519 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:28.703514 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.703477 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" event={"ID":"06f128ca-a03d-471b-a951-4d19cddd1785","Type":"ContainerStarted","Data":"634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d"} Nov 02 13:58:28.703715 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.703527 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" event={"ID":"06f128ca-a03d-471b-a951-4d19cddd1785","Type":"ContainerStarted","Data":"84ed00777f8bbd9254371eddf06d7e66fd2097e940c6d1ee1ac4c6f6d6375f7c"} Nov 02 13:58:28.752611 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:28.752500 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podStartSLOduration=1.752485614 podStartE2EDuration="1.752485614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:28.751711949 +0000 UTC m=+857.477333718" watchObservedRunningTime="2026-11-02 13:58:28.752485614 +0000 UTC m=+857.478107385" Nov 02 13:58:29.183524 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:29.183489 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:29.185161 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:29.185132 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 13:58:29.185281 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:29.185184 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 13:58:29.281433 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:29.281393 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:30.184078 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:30.184037 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 13:58:30.184576 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:30.184115 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 13:58:31.011631 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:31.011552 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:31.081798 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:58:31.081762 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:58:31.184282 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:31.184248 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 13:58:31.184679 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:31.184300 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 13:58:31.401835 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:31.401784 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:31.571183 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:31.571133 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:31.642042 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:31.641763 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:58:32.184463 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:32.184422 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 13:58:32.184899 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:32.184497 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 13:58:32.818586 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:32.818546 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:33.184577 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:33.184538 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 13:58:33.185134 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:33.184605 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 13:58:34.183968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:34.183928 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 13:58:34.184211 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:34.184016 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 13:58:34.563025 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:34.562916 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:35.184118 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:35.184080 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 13:58:35.184296 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:35.184156 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 13:58:36.229025 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:36.228969 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:36.229892 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:36.229862 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:36.230362 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:36.230330 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 13:58:36.230918 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:36.230893 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:38.084727 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:38.084687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:39.847035 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:39.844237 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:40.119199 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:40.119113 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bf89bf644-7466n"] Nov 02 13:58:40.216465 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:40.216426 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:40.421111 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:40.421077 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:42.317009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.289742 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj"] Nov 02 13:58:42.317009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.310363 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:42.317009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.316574 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj"] Nov 02 13:58:42.317009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.316679 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.352253 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.352218 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"distinct-2-w2-0969b1-kserve-self-signed-certs\"" Nov 02 13:58:42.466344 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.466177 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-dshm\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.466344 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.466246 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldlkz\" (UniqueName: \"kubernetes.io/projected/2a01bbb4-da32-470c-8ba4-54ac28a79c19-kube-api-access-ldlkz\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.466344 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.466312 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tls-certs\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.466674 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.466366 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-model-cache\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.466674 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.466423 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-home\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.466674 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.466449 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tmp-dir\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.571607 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.571021 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-home\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.572366 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.571937 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tmp-dir\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.572613 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.572593 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-home\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.572750 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.572304 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tmp-dir\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.573118 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.573097 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-dshm\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.590825 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.590792 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldlkz\" (UniqueName: \"kubernetes.io/projected/2a01bbb4-da32-470c-8ba4-54ac28a79c19-kube-api-access-ldlkz\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.605287 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.591709 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tls-certs\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.606102 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.606079 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-model-cache\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.609012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.606834 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-model-cache\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.609169 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.607364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:42.609435 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.607766 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-dshm\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.621454 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.621415 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tls-certs\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.647766 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.647733 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldlkz\" (UniqueName: \"kubernetes.io/projected/2a01bbb4-da32-470c-8ba4-54ac28a79c19-kube-api-access-ldlkz\") pod \"distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:42.684354 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:42.683772 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:43.180079 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:43.179780 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj"] Nov 02 13:58:43.208018 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:58:43.207970 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a01bbb4_da32_470c_8ba4_54ac28a79c19.slice/crio-5ddcce4b9cd5e983cf92cf1bf0684b8e730fdbab46bf1ade0d17fb6272877dd8 WatchSource:0}: Error finding container 5ddcce4b9cd5e983cf92cf1bf0684b8e730fdbab46bf1ade0d17fb6272877dd8: Status 404 returned error can't find the container with id 5ddcce4b9cd5e983cf92cf1bf0684b8e730fdbab46bf1ade0d17fb6272877dd8 Nov 02 13:58:43.910722 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:43.910689 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" event={"ID":"2a01bbb4-da32-470c-8ba4-54ac28a79c19","Type":"ContainerStarted","Data":"d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708"} Nov 02 13:58:43.911437 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:43.911231 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" event={"ID":"2a01bbb4-da32-470c-8ba4-54ac28a79c19","Type":"ContainerStarted","Data":"5ddcce4b9cd5e983cf92cf1bf0684b8e730fdbab46bf1ade0d17fb6272877dd8"} Nov 02 13:58:43.970020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:43.969631 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podStartSLOduration=1.9696116190000001 podStartE2EDuration="1.969611619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:43.968347216 +0000 UTC m=+872.693968989" watchObservedRunningTime="2026-11-02 13:58:43.969611619 +0000 UTC m=+872.695233392" Nov 02 13:58:44.665594 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:44.657862 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:44.685304 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:44.685269 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:45.120189 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.120096 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" podUID="c5576506-ec06-4575-8038-cd2e6fda0c07" containerName="payload-processing" containerID="cri-o://7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a" gracePeriod=25 Nov 02 13:58:45.388645 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.388618 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:58:45.452879 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.452849 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-provider-credentials\") pod \"c5576506-ec06-4575-8038-cd2e6fda0c07\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " Nov 02 13:58:45.453103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.453013 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-routing-overlay-volume\") pod \"c5576506-ec06-4575-8038-cd2e6fda0c07\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " Nov 02 13:58:45.453103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.453098 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-kube-api-access-45cnp\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-kube-api-access-45cnp\") pod \"c5576506-ec06-4575-8038-cd2e6fda0c07\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " Nov 02 13:58:45.453219 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.453132 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-plugins-config-volume\") pod \"c5576506-ec06-4575-8038-cd2e6fda0c07\" (UID: \"c5576506-ec06-4575-8038-cd2e6fda0c07\") " Nov 02 13:58:45.453427 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.453401 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-routing-overlay-volume" pod "c5576506-ec06-4575-8038-cd2e6fda0c07" (UID: "c5576506-ec06-4575-8038-cd2e6fda0c07"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:45.453711 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.453689 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:58:45.453826 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.453742 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-plugins-config-volume" pod "c5576506-ec06-4575-8038-cd2e6fda0c07" (UID: "c5576506-ec06-4575-8038-cd2e6fda0c07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:45.455236 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.455202 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-provider-credentials" pod "c5576506-ec06-4575-8038-cd2e6fda0c07" (UID: "c5576506-ec06-4575-8038-cd2e6fda0c07"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:45.455497 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.455477 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-kube-api-access-45cnp" pod "c5576506-ec06-4575-8038-cd2e6fda0c07" (UID: "c5576506-ec06-4575-8038-cd2e6fda0c07"). InnerVolumeSpecName "kube-api-access-45cnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:45.555407 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.555368 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45cnp\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-kube-api-access-45cnp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:58:45.555407 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.555404 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5576506-ec06-4575-8038-cd2e6fda0c07-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:58:45.555614 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.555418 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5576506-ec06-4575-8038-cd2e6fda0c07-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 13:58:45.922194 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.922159 5591 generic.go:356] "Generic (PLEG): container finished" podID="c5576506-ec06-4575-8038-cd2e6fda0c07" containerID="7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a" exitCode=0 Nov 02 13:58:45.922382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.922231 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" Nov 02 13:58:45.922382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.922248 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" event={"ID":"c5576506-ec06-4575-8038-cd2e6fda0c07","Type":"ContainerDied","Data":"7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a"} Nov 02 13:58:45.922382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.922334 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bf89bf644-7466n" event={"ID":"c5576506-ec06-4575-8038-cd2e6fda0c07","Type":"ContainerDied","Data":"b6dda2c1cd7c032f5a517b70578dec8b799926304171e3bfe022a4dc565f4f97"} Nov 02 13:58:45.922382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.922362 5591 scope.go:122] "RemoveContainer" containerID="7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a" Nov 02 13:58:45.932124 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.932104 5591 scope.go:122] "RemoveContainer" containerID="7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a" Nov 02 13:58:45.932380 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:58:45.932360 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a\": container with ID starting with 7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a not found: ID does not exist" containerID="7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a" Nov 02 13:58:45.932441 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.932393 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a"} err="failed to get container status \"7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a\": rpc error: code = NotFound desc = could not find container \"7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a\": container with ID starting with 7992cf0f8ac82c7cb61e1f7cbbdcded2acc92cfae382b6720066f51fa897479a not found: ID does not exist" Nov 02 13:58:45.981519 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.981485 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bf89bf644-7466n"] Nov 02 13:58:45.987032 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:45.987002 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bf89bf644-7466n"] Nov 02 13:58:46.515663 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:46.515550 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:47.429942 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:47.429908 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5576506-ec06-4575-8038-cd2e6fda0c07" path="/var/lib/kubelet/pods/c5576506-ec06-4575-8038-cd2e6fda0c07/volumes" Nov 02 13:58:47.862840 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:47.862742 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:48.415432 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:48.415383 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:50.270580 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:50.270539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:50.346671 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:50.346634 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:58:52.675714 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:52.675674 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:52.686506 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:52.686474 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:53.022909 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:53.022771 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:53.093960 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:53.093679 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 13:58:54.878127 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.877763 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:54.967659 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.967621 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv"] Nov 02 13:58:54.968160 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.968146 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c5576506-ec06-4575-8038-cd2e6fda0c07" containerName="payload-processing" Nov 02 13:58:54.968213 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.968161 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5576506-ec06-4575-8038-cd2e6fda0c07" containerName="payload-processing" Nov 02 13:58:54.968255 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.968245 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5576506-ec06-4575-8038-cd2e6fda0c07" containerName="payload-processing" Nov 02 13:58:54.973144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.973096 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:54.976174 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.976154 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4"] Nov 02 13:58:54.979803 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.979781 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:54.989137 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:54.989108 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv"] Nov 02 13:58:55.020130 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.020095 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4"] Nov 02 13:58:55.036641 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.036602 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.036950 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.036926 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.037213 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.037188 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4251524d-6cb3-430f-930f-02bc806b91f9-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.037366 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.037251 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4251524d-6cb3-430f-930f-02bc806b91f9-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.037366 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.037280 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.037366 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.037309 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxjll\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-kube-api-access-vxjll\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.037814 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.037382 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.037814 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.037432 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.037814 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.037488 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.138638 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138612 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.138798 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138647 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.138798 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138669 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.138798 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138740 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4251524d-6cb3-430f-930f-02bc806b91f9-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.138798 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138773 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.138959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138817 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4251524d-6cb3-430f-930f-02bc806b91f9-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.138959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138839 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.138959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138857 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxjll\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-kube-api-access-vxjll\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.138959 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138916 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.139166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.138970 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.139166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139151 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.139245 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139175 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.139291 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139265 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.139337 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139296 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.139385 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139336 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.139385 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139370 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.139497 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139475 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.139557 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139516 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.139557 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139544 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4679\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-kube-api-access-q4679\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.139669 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139557 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4251524d-6cb3-430f-930f-02bc806b91f9-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.139669 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139572 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.139780 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139676 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74abb745-c01a-4429-b857-9d4cfd7f2b95-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.139826 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.139802 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.141588 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.141564 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.141694 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.141593 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4251524d-6cb3-430f-930f-02bc806b91f9-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.163012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.162972 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.163812 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.163788 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxjll\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-kube-api-access-vxjll\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.241124 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241083 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241296 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241167 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241364 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241330 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4679\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-kube-api-access-q4679\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241433 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241395 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74abb745-c01a-4429-b857-9d4cfd7f2b95-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241513 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241496 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241571 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241549 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241627 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241559 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241683 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241626 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241683 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241669 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241784 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241700 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241864 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241839 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.241952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.241938 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.242126 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.242106 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.242216 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.242200 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74abb745-c01a-4429-b857-9d4cfd7f2b95-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.243673 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.243650 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.243947 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.243929 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.271402 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.271379 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.271767 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.271730 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4679\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-kube-api-access-q4679\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.287113 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.287087 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:55.297424 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.297401 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:55.617103 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.617069 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv"] Nov 02 13:58:55.630721 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:58:55.630685 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4251524d_6cb3_430f_930f_02bc806b91f9.slice/crio-384788ac1a6677f3a0387d9f803e8409d9deb4c8ab4ccd5b1113e0fe236e5c9d WatchSource:0}: Error finding container 384788ac1a6677f3a0387d9f803e8409d9deb4c8ab4ccd5b1113e0fe236e5c9d: Status 404 returned error can't find the container with id 384788ac1a6677f3a0387d9f803e8409d9deb4c8ab4ccd5b1113e0fe236e5c9d Nov 02 13:58:55.632919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.632887 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:55.633031 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.632969 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:55.633090 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.633030 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:55.671531 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.671503 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4"] Nov 02 13:58:55.674746 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.674715 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:55.674836 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.674776 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:55.674836 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:55.674804 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:58:56.054724 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.054553 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" event={"ID":"4251524d-6cb3-430f-930f-02bc806b91f9","Type":"ContainerStarted","Data":"e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7"} Nov 02 13:58:56.054724 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.054613 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" event={"ID":"4251524d-6cb3-430f-930f-02bc806b91f9","Type":"ContainerStarted","Data":"384788ac1a6677f3a0387d9f803e8409d9deb4c8ab4ccd5b1113e0fe236e5c9d"} Nov 02 13:58:56.056318 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.056289 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" event={"ID":"74abb745-c01a-4429-b857-9d4cfd7f2b95","Type":"ContainerStarted","Data":"a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036"} Nov 02 13:58:56.056443 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.056336 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" event={"ID":"74abb745-c01a-4429-b857-9d4cfd7f2b95","Type":"ContainerStarted","Data":"5cde715fce2e8ef3b7d5cc022b074dc8d6b0b5cf7ce6b68145a204072cefb379"} Nov 02 13:58:56.188069 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.187951 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podStartSLOduration=2.187927756 podStartE2EDuration="2.187927756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:56.132393445 +0000 UTC m=+884.858015245" watchObservedRunningTime="2026-11-02 13:58:56.187927756 +0000 UTC m=+884.913549529" Nov 02 13:58:56.287425 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.287382 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:58:56.289070 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.289046 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 13:58:56.289164 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.289102 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 13:58:56.297710 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.297681 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:58:56.298910 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.298890 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 13:58:56.299021 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.298937 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 13:58:56.949023 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:56.945141 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:57.287616 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:57.287517 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 13:58:57.287616 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:57.287585 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 13:58:57.298063 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:57.298032 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 13:58:57.298211 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:57.298088 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 13:58:58.288109 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:58.288069 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 13:58:58.288607 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:58.288140 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 13:58:58.298368 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:58.298340 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 13:58:58.298483 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:58.298384 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 13:58:59.194635 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:59.194539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:59.288388 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:59.288348 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 13:58:59.288801 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:59.288420 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 13:58:59.298563 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:59.298533 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 13:58:59.298698 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:58:59.298583 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 13:59:00.288205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:00.288169 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 13:59:00.288418 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:00.288228 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 13:59:00.298112 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:00.298084 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 13:59:00.298226 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:00.298130 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 13:59:01.104311 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:59:01.104270 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:59:01.291006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:01.288321 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 13:59:01.291006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:01.288384 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 13:59:01.303063 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:01.299424 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 13:59:01.303063 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:01.299490 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 13:59:01.303063 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:01.300521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:02.287807 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:02.287769 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 13:59:02.288046 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:02.287850 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 13:59:02.298233 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:02.298202 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 13:59:02.298676 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:02.298643 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 13:59:03.293034 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:03.292974 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:59:03.293463 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:03.293433 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:59:03.294227 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:03.294208 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 13:59:03.302376 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:03.302357 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:59:03.302747 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:03.302592 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:59:03.303501 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:03.303483 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 13:59:03.368020 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:03.367935 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podStartSLOduration=9.367916815 podStartE2EDuration="9.367916815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:56.190330914 +0000 UTC m=+884.915952684" watchObservedRunningTime="2026-11-02 13:59:03.367916815 +0000 UTC m=+892.093538587" Nov 02 13:59:03.567603 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:03.567507 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:05.481389 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:05.481350 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:06.087847 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:06.087808 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:07.162822 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:07.162785 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:07.331533 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:07.331500 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:08.990470 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:08.990351 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:10.797002 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:10.796742 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:11.592733 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:11.592701 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:59:11.595204 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:11.595179 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 13:59:12.615539 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.612470 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:12.662663 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.662630 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv"] Nov 02 13:59:12.668104 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.668079 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.706216 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.706186 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv"] Nov 02 13:59:12.736291 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736254 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.736618 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736319 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.736618 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736506 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.736618 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736566 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.736861 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736693 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.736861 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736722 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.736861 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736738 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdp4\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-kube-api-access-qsdp4\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.736861 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736768 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.737068 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.736975 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2fb505-7f16-449b-9b80-ae516cf2bba7-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.838744 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.838710 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.838902 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.838753 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.838902 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.838803 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.838902 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.838819 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.838902 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.838834 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdp4\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-kube-api-access-qsdp4\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.838902 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.838870 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.839186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.838928 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2fb505-7f16-449b-9b80-ae516cf2bba7-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.839186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.838976 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.839186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.839033 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.839186 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.839177 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.839394 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.839251 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.839451 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.839397 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.839721 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.839694 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2fb505-7f16-449b-9b80-ae516cf2bba7-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.839799 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.839727 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.841207 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.841186 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.841346 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.841329 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.869207 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.869143 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdp4\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-kube-api-access-qsdp4\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.869663 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.869643 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:12.931088 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.931053 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:59:12.981826 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:12.981795 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:13.026090 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.026049 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf"] Nov 02 13:59:13.031037 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.031010 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.059036 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.059003 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf"] Nov 02 13:59:13.143589 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.143555 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.143589 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.143590 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e739adc0-9202-4502-846d-089cb272b6db-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.143807 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.143610 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.143807 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.143713 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.143807 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.143744 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e739adc0-9202-4502-846d-089cb272b6db-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.143807 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.143769 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdhr\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-kube-api-access-gjdhr\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.144021 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.143974 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.144093 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.144030 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.144093 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.144077 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.245733 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.245702 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.245733 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.245737 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.245968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.245768 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.245968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.245848 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.245968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.245864 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e739adc0-9202-4502-846d-089cb272b6db-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.245968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.245885 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.245968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.245920 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.245968 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.245941 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e739adc0-9202-4502-846d-089cb272b6db-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.246281 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.246162 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.246281 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.246205 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdhr\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-kube-api-access-gjdhr\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.246388 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.246324 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.246438 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.246390 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.246499 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.246482 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.246772 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.246748 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e739adc0-9202-4502-846d-089cb272b6db-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.248301 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.248281 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.248595 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.248578 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e739adc0-9202-4502-846d-089cb272b6db-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.277077 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.277049 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.277196 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.277160 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdhr\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-kube-api-access-gjdhr\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.312381 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.312360 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv"] Nov 02 13:59:13.314368 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:59:13.314341 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2fb505_7f16_449b_9b80_ae516cf2bba7.slice/crio-eb56b4679fcb8b1eeffd20f9886422e49e5f6ffbe666ad73ef9937b1f179a297 WatchSource:0}: Error finding container eb56b4679fcb8b1eeffd20f9886422e49e5f6ffbe666ad73ef9937b1f179a297: Status 404 returned error can't find the container with id eb56b4679fcb8b1eeffd20f9886422e49e5f6ffbe666ad73ef9937b1f179a297 Nov 02 13:59:13.316753 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.316719 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:13.316844 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.316791 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:13.316844 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.316819 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:13.369221 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.369196 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:13.729924 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.729893 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf"] Nov 02 13:59:13.731706 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:59:13.731677 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode739adc0_9202_4502_846d_089cb272b6db.slice/crio-04c18cfd25b8fdae77870fe62aed2331e33d1ec4fbe2dda56811f9ff65bdd4fb WatchSource:0}: Error finding container 04c18cfd25b8fdae77870fe62aed2331e33d1ec4fbe2dda56811f9ff65bdd4fb: Status 404 returned error can't find the container with id 04c18cfd25b8fdae77870fe62aed2331e33d1ec4fbe2dda56811f9ff65bdd4fb Nov 02 13:59:13.734442 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.734403 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:13.734536 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.734499 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:13.734605 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:13.734545 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:14.032058 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.027584 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:14.232907 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.224464 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" event={"ID":"e739adc0-9202-4502-846d-089cb272b6db","Type":"ContainerStarted","Data":"575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf"} Nov 02 13:59:14.232907 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.224551 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" event={"ID":"e739adc0-9202-4502-846d-089cb272b6db","Type":"ContainerStarted","Data":"04c18cfd25b8fdae77870fe62aed2331e33d1ec4fbe2dda56811f9ff65bdd4fb"} Nov 02 13:59:14.232907 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.227954 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" event={"ID":"be2fb505-7f16-449b-9b80-ae516cf2bba7","Type":"ContainerStarted","Data":"c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81"} Nov 02 13:59:14.232907 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.228047 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" event={"ID":"be2fb505-7f16-449b-9b80-ae516cf2bba7","Type":"ContainerStarted","Data":"eb56b4679fcb8b1eeffd20f9886422e49e5f6ffbe666ad73ef9937b1f179a297"} Nov 02 13:59:14.374062 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.373105 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:14.374062 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.373660 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 13:59:14.374062 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.373711 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 13:59:14.396005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.387513 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podStartSLOduration=2.387490589 podStartE2EDuration="2.387490589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:14.316445291 +0000 UTC m=+903.042067086" watchObservedRunningTime="2026-11-02 13:59:14.387490589 +0000 UTC m=+903.113112360" Nov 02 13:59:14.404005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.401658 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podStartSLOduration=2.401643751 podStartE2EDuration="2.401643751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:14.390450743 +0000 UTC m=+903.116072534" watchObservedRunningTime="2026-11-02 13:59:14.401643751 +0000 UTC m=+903.127265523" Nov 02 13:59:14.995008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.994220 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:14.995008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.994417 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 13:59:14.995008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:14.994477 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 13:59:15.224008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:15.223101 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:15.370494 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:15.370398 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 13:59:15.370776 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:15.370742 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 13:59:15.988707 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:15.988395 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 13:59:15.988707 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:15.988463 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 13:59:16.381065 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:16.380766 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 13:59:16.381065 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:16.380832 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 13:59:16.982480 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:16.982437 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 13:59:16.982670 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:16.982507 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 13:59:17.374482 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:17.374383 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 13:59:17.374775 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:17.374744 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 13:59:17.519008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:17.516243 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:17.982605 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:17.982552 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 13:59:17.982824 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:17.982624 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 13:59:18.371482 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:18.371377 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 13:59:18.371773 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:18.371742 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 13:59:18.982484 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:18.982440 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 13:59:18.982903 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:18.982507 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 13:59:19.374924 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:19.374400 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 13:59:19.374924 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:19.374469 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 13:59:19.683600 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:19.683557 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:19.983200 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:19.983095 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 13:59:19.983200 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:19.983170 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 13:59:20.370691 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:20.370600 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 13:59:20.370691 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:20.370661 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 13:59:20.735956 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:20.735913 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:20.982380 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:20.982341 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 13:59:20.982586 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:20.982408 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 13:59:21.147259 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:21.147218 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:21.369671 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:21.369628 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 13:59:21.369844 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:21.369702 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 13:59:21.795728 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:21.795691 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:21.988720 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:21.988686 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:21.989033 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:21.989015 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:21.989767 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:21.989745 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 13:59:22.374898 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:22.374866 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:22.375399 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:22.375174 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:22.375911 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:22.375886 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 13:59:23.583127 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:23.582769 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:25.167413 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:25.167361 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:26.874849 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:26.874811 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:28.786003 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:28.783911 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:30.544889 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:30.544853 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:31.352593 ip-10-0-140-236 kubenswrapper[5591]: E1102 13:59:31.352523 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 13:59:32.397012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:32.394076 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 13:59:32.478007 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:32.477083 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:32.596510 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:32.596458 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:32.640052 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:32.630753 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:32.668174 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:32.668042 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:32.819010 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:32.815112 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:33.040222 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.040115 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:33.397393 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.397301 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 13:59:33.398382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.398359 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:59:33.480219 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.480176 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:33.480999 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.480966 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:59:33.500944 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.500916 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 13:59:33.518637 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.518300 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:33.608999 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.581142 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 13:59:33.948546 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:33.941545 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:34.586962 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.586924 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j"] Nov 02 13:59:34.592406 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.592377 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.626677 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.626643 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j"] Nov 02 13:59:34.755349 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.755310 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.755349 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.755350 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.755586 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.755371 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6x7\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-kube-api-access-hm6x7\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.755634 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.755582 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.755672 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.755657 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.755720 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.755687 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.755804 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.755786 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.756052 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.756031 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.756122 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.756104 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.857737 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.857655 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.857737 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.857706 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.857930 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.857760 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.857930 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.857799 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.857930 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.857817 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6x7\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-kube-api-access-hm6x7\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.857930 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.857836 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.858164 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.858040 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.858164 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.858100 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.858263 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.858186 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.858263 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.858208 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.858263 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.858242 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.858399 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.858379 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.858588 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.858565 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.858649 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.858603 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.860274 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.860250 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.860429 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.860410 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.882257 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.882234 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.886316 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.886290 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6x7\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-kube-api-access-hm6x7\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:34.910679 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:34.910646 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:35.229705 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:35.229671 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j"] Nov 02 13:59:35.230411 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:59:35.230386 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6c7bf0_4b50_459f_9265_3ef9336f2e03.slice/crio-ce02aaf13c296d12d70c32305d06e74df7131743f2f76db4c275c144dafce534 WatchSource:0}: Error finding container ce02aaf13c296d12d70c32305d06e74df7131743f2f76db4c275c144dafce534: Status 404 returned error can't find the container with id ce02aaf13c296d12d70c32305d06e74df7131743f2f76db4c275c144dafce534 Nov 02 13:59:35.232715 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:35.232680 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:35.232795 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:35.232748 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:35.232839 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:35.232797 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:35.491513 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:35.491416 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" event={"ID":"df6c7bf0-4b50-459f-9265-3ef9336f2e03","Type":"ContainerStarted","Data":"2eb2d6e0c6d3d5e920868f3e5d9b96bcfba26359e03f226548c4792e83b60e67"} Nov 02 13:59:35.491513 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:35.491477 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" event={"ID":"df6c7bf0-4b50-459f-9265-3ef9336f2e03","Type":"ContainerStarted","Data":"ce02aaf13c296d12d70c32305d06e74df7131743f2f76db4c275c144dafce534"} Nov 02 13:59:35.591433 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:35.591381 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podStartSLOduration=1.591367475 podStartE2EDuration="1.591367475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:35.589303953 +0000 UTC m=+924.314925726" watchObservedRunningTime="2026-11-02 13:59:35.591367475 +0000 UTC m=+924.316989245" Nov 02 13:59:35.911497 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:35.911458 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:36.191847 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:36.191735 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:36.912766 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:36.912731 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 13:59:36.913195 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:36.912784 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 13:59:37.862497 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:37.862416 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:37.913153 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:37.912689 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 13:59:37.913153 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:37.912756 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 13:59:38.911072 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:38.911035 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 13:59:38.911261 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:38.911119 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 13:59:39.517047 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:39.516832 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:39.911368 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:39.911325 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 13:59:39.911595 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:39.911393 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 13:59:40.279274 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:40.279187 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:40.911044 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:40.911006 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 13:59:40.911472 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:40.911063 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 13:59:41.149466 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:41.149428 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:41.911546 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:41.911510 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 13:59:41.912121 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:41.911565 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 13:59:42.011547 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.011506 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 13:59:42.821009 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.817040 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z"] Nov 02 13:59:42.859643 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.859605 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z"] Nov 02 13:59:42.902175 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.860973 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949316 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949373 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949421 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949502 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79aa1db-ad41-4b56-87a5-74858aa385ce-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949532 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949628 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949761 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949830 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8lwq\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-kube-api-access-q8lwq\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:42.950144 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:42.949858 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.108555 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.106279 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.127445 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8lwq\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-kube-api-access-q8lwq\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.127685 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.127779 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.127812 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.128068 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.128157 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79aa1db-ad41-4b56-87a5-74858aa385ce-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.128196 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.128511 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.128659 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.139512 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.153005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.140958 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.197044 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.196785 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.199919 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.199884 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79aa1db-ad41-4b56-87a5-74858aa385ce-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.200642 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.200616 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.201095 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.201071 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.253965 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.253930 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.257603 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.257542 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.261526 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.261499 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8lwq\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-kube-api-access-q8lwq\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.272486 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.270944 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:43.701347 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.701311 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z"] Nov 02 13:59:43.792036 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:59:43.791999 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79aa1db_ad41_4b56_87a5_74858aa385ce.slice/crio-fbd00c118efb3d591875f66915282e5413e8ee2856d22c47bda02c9dd232e178 WatchSource:0}: Error finding container fbd00c118efb3d591875f66915282e5413e8ee2856d22c47bda02c9dd232e178: Status 404 returned error can't find the container with id fbd00c118efb3d591875f66915282e5413e8ee2856d22c47bda02c9dd232e178 Nov 02 13:59:43.805302 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.805256 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:43.805547 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.805521 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:43.805741 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:43.805707 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:44.042063 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:44.031088 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:59:44.042063 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:44.031176 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:44.669871 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:44.669084 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:44.727016 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:44.722494 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" event={"ID":"b79aa1db-ad41-4b56-87a5-74858aa385ce","Type":"ContainerStarted","Data":"5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926"} Nov 02 13:59:44.727016 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:44.722568 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" event={"ID":"b79aa1db-ad41-4b56-87a5-74858aa385ce","Type":"ContainerStarted","Data":"fbd00c118efb3d591875f66915282e5413e8ee2856d22c47bda02c9dd232e178"} Nov 02 13:59:44.779673 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:44.779610 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podStartSLOduration=2.779590611 podStartE2EDuration="2.779590611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:44.775421727 +0000 UTC m=+933.501043499" watchObservedRunningTime="2026-11-02 13:59:44.779590611 +0000 UTC m=+933.505212387" Nov 02 13:59:45.077006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.073139 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:59:45.077006 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.073210 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:59:45.079848 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.079802 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:45.172246 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.172213 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:45.172535 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.172503 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:45.173213 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.173193 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 13:59:45.271674 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.271645 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:45.273174 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.273145 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 13:59:45.273317 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:45.273206 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 13:59:46.272555 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:46.272516 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 13:59:46.272930 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:46.272582 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 13:59:47.033007 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:47.025474 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:47.272066 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:47.272028 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 13:59:47.272253 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:47.272092 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 13:59:48.272123 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:48.272083 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 13:59:48.272638 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:48.272134 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 13:59:48.796086 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:48.796051 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:48.963008 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:48.962441 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:49.272087 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:49.272051 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 13:59:49.272252 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:49.272114 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 13:59:50.272012 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:50.271943 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 13:59:50.272205 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:50.272041 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 13:59:50.803777 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:50.803739 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:51.272316 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:51.272279 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 13:59:51.272523 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:51.272352 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 13:59:52.276374 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:52.276342 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:52.276754 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:52.276601 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:52.277346 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:52.277330 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 13:59:52.560620 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:52.560523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:54.256958 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:54.253883 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:54.589005 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:54.588891 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:55.113241 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.113198 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9"] Nov 02 13:59:55.118140 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.118121 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.153935 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.153895 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9"] Nov 02 13:59:55.196320 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196284 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.196522 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196412 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.196522 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196480 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a6917-c963-471b-8f38-ad047570afb1-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.196605 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196581 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz42l\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-kube-api-access-gz42l\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.196640 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196620 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.196676 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196663 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.196744 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196729 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7a6917-c963-471b-8f38-ad047570afb1-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.196794 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196754 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.196842 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.196834 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.298578 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.298542 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299027 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.298632 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299027 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.298706 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a6917-c963-471b-8f38-ad047570afb1-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299027 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.298743 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz42l\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-kube-api-access-gz42l\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299027 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.298782 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299027 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.298838 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299027 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.298873 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7a6917-c963-471b-8f38-ad047570afb1-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299027 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.298906 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299422 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.299028 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299422 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.299112 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299422 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.299154 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299422 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.299315 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299422 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.299384 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.299659 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.299614 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a6917-c963-471b-8f38-ad047570afb1-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.301130 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.301105 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.301452 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.301431 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7a6917-c963-471b-8f38-ad047570afb1-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.325323 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.325301 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.325444 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.325384 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz42l\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-kube-api-access-gz42l\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.431499 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.431470 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:55.752166 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.752138 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9"] Nov 02 13:59:55.753610 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:59:55.753584 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7a6917_c963_471b_8f38_ad047570afb1.slice/crio-0bffbcbd26ec72cef5e08722b127e66589acf4dacf6bfa6bcd1d7442c65518d5 WatchSource:0}: Error finding container 0bffbcbd26ec72cef5e08722b127e66589acf4dacf6bfa6bcd1d7442c65518d5: Status 404 returned error can't find the container with id 0bffbcbd26ec72cef5e08722b127e66589acf4dacf6bfa6bcd1d7442c65518d5 Nov 02 13:59:55.755890 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.755857 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:55.755952 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.755927 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:55.756007 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.755958 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:55.806710 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.806675 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" event={"ID":"6b7a6917-c963-471b-8f38-ad047570afb1","Type":"ContainerStarted","Data":"0bffbcbd26ec72cef5e08722b127e66589acf4dacf6bfa6bcd1d7442c65518d5"} Nov 02 13:59:55.970390 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:55.964685 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:56.813704 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:56.813661 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" event={"ID":"6b7a6917-c963-471b-8f38-ad047570afb1","Type":"ContainerStarted","Data":"46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15"} Nov 02 13:59:56.898461 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:56.898410 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podStartSLOduration=1.8983932399999999 podStartE2EDuration="1.89839324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:56.885745314 +0000 UTC m=+945.611367084" watchObservedRunningTime="2026-11-02 13:59:56.89839324 +0000 UTC m=+945.624015072" Nov 02 13:59:57.451098 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:57.451060 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 13:59:57.456631 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:57.453807 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 13:59:57.456631 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:57.453898 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 13:59:57.664590 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:57.664510 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:58.177240 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.177200 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6"] Nov 02 13:59:58.181183 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.181165 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.218964 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.218933 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6"] Nov 02 13:59:58.266070 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.265735 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.266070 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.265885 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.266070 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.265915 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.266070 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.265947 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.266434 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.266123 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.266434 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.266153 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5454e3c0-401e-4a75-9602-729f031c98d6-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.266434 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.266208 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbqns\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-kube-api-access-zbqns\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.266434 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.266248 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5454e3c0-401e-4a75-9602-729f031c98d6-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.266434 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.266280 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.367938 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.367904 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5454e3c0-401e-4a75-9602-729f031c98d6-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368161 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.367974 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbqns\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-kube-api-access-zbqns\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368161 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368043 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5454e3c0-401e-4a75-9602-729f031c98d6-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368161 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368072 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368161 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368147 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368270 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368304 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368382 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368356 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368538 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368444 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368624 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368602 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.368823 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.368799 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.369156 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.369128 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.369425 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.369405 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.369552 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.369419 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5454e3c0-401e-4a75-9602-729f031c98d6-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.370751 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.370727 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.371605 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.371582 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5454e3c0-401e-4a75-9602-729f031c98d6-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.393769 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.393734 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.393895 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.393801 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbqns\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-kube-api-access-zbqns\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.454228 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.453705 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 13:59:58.454228 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.453767 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 13:59:58.495723 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.495694 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 13:59:58.857058 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.856322 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6"] Nov 02 13:59:58.861748 ip-10-0-140-236 kubenswrapper[5591]: W1102 13:59:58.861720 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5454e3c0_401e_4a75_9602_729f031c98d6.slice/crio-1a8de16bd7fe0ae7cefc20d91d25c723201013008a2ee38a06dc4c83119bcd92 WatchSource:0}: Error finding container 1a8de16bd7fe0ae7cefc20d91d25c723201013008a2ee38a06dc4c83119bcd92: Status 404 returned error can't find the container with id 1a8de16bd7fe0ae7cefc20d91d25c723201013008a2ee38a06dc4c83119bcd92 Nov 02 13:59:58.864363 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.864333 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:58.864447 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.864398 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:58.864447 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:58.864427 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:59:59.448150 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:59.448111 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 13:59:59.448757 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:59.448720 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 13:59:59.470701 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:59.470665 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:59.795832 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:59.791657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:59.875073 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:59.875034 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" event={"ID":"5454e3c0-401e-4a75-9602-729f031c98d6","Type":"ContainerStarted","Data":"8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54"} Nov 02 13:59:59.875358 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:59.875338 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" event={"ID":"5454e3c0-401e-4a75-9602-729f031c98d6","Type":"ContainerStarted","Data":"1a8de16bd7fe0ae7cefc20d91d25c723201013008a2ee38a06dc4c83119bcd92"} Nov 02 13:59:59.987050 ip-10-0-140-236 kubenswrapper[5591]: I1102 13:59:59.968467 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podStartSLOduration=1.968447033 podStartE2EDuration="1.968447033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:59.942151107 +0000 UTC m=+948.667772878" watchObservedRunningTime="2026-11-02 13:59:59.968447033 +0000 UTC m=+948.694068805" Nov 02 14:00:00.202750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.202704 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j"] Nov 02 14:00:00.206746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.206723 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:00.255066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.240143 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\"" Nov 02 14:00:00.265108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.265053 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh"] Nov 02 14:00:00.409008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.408891 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e82b09f-0e50-468c-a261-284d2407daa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-66v9j\" (UID: \"1e82b09f-0e50-468c-a261-284d2407daa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:00.409008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.408961 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mm5\" (UniqueName: \"kubernetes.io/projected/1e82b09f-0e50-468c-a261-284d2407daa4-kube-api-access-99mm5\") pod \"maas-api-key-cleanup-29893800-66v9j\" (UID: \"1e82b09f-0e50-468c-a261-284d2407daa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:00.555012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.554896 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e82b09f-0e50-468c-a261-284d2407daa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-66v9j\" (UID: \"1e82b09f-0e50-468c-a261-284d2407daa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:00.559942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.559910 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mm5\" (UniqueName: \"kubernetes.io/projected/1e82b09f-0e50-468c-a261-284d2407daa4-kube-api-access-99mm5\") pod \"maas-api-key-cleanup-29893800-66v9j\" (UID: \"1e82b09f-0e50-468c-a261-284d2407daa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:00.567105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.560976 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 14:00:00.567105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.561048 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 14:00:00.570080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.568561 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:00.570080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.568618 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:00.578898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.578869 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e82b09f-0e50-468c-a261-284d2407daa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-66v9j\" (UID: \"1e82b09f-0e50-468c-a261-284d2407daa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:00.593320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.593283 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:00.607031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.594006 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j"] Nov 02 14:00:00.607031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.594079 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh"] Nov 02 14:00:00.607031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.594120 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 14:00:00.687006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.683209 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db249407-60c6-4de9-8811-6e401a260bbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh\" (UID: \"db249407-60c6-4de9-8811-6e401a260bbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:00.687006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.683423 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcfz\" (UniqueName: \"kubernetes.io/projected/db249407-60c6-4de9-8811-6e401a260bbf-kube-api-access-zzcfz\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh\" (UID: \"db249407-60c6-4de9-8811-6e401a260bbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:00.731538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.731506 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:00.742011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.735719 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mm5\" (UniqueName: \"kubernetes.io/projected/1e82b09f-0e50-468c-a261-284d2407daa4-kube-api-access-99mm5\") pod \"maas-api-key-cleanup-29893800-66v9j\" (UID: \"1e82b09f-0e50-468c-a261-284d2407daa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:00.796009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.790786 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcfz\" (UniqueName: \"kubernetes.io/projected/db249407-60c6-4de9-8811-6e401a260bbf-kube-api-access-zzcfz\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh\" (UID: \"db249407-60c6-4de9-8811-6e401a260bbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:00.796009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.791027 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db249407-60c6-4de9-8811-6e401a260bbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh\" (UID: \"db249407-60c6-4de9-8811-6e401a260bbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:00.796009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.792055 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db249407-60c6-4de9-8811-6e401a260bbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh\" (UID: \"db249407-60c6-4de9-8811-6e401a260bbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:00.825278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.825191 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:00.870242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.851965 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcfz\" (UniqueName: \"kubernetes.io/projected/db249407-60c6-4de9-8811-6e401a260bbf-kube-api-access-zzcfz\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh\" (UID: \"db249407-60c6-4de9-8811-6e401a260bbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:00.929568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:00.929507 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:01.091054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.086302 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:01.303276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.303227 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:01.474183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.473486 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 14:00:01.474183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.473556 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 14:00:01.547007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.535874 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:01.547007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.535934 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:01.558508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.558457 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:01.559235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.559083 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:01.582703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.582662 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:00:01.595483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.595421 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:01.615848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.615799 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:01.628243 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:01.628202 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:00:01.668078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.668031 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:00:01.738304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:01.738140 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:02.104016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.058091 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:02.466419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.418424 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:00:02.466419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.418881 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:00:02.515428 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.515391 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 14:00:02.519135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.519078 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 14:00:02.584063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.559097 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:02.619135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.619085 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:02.619377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.565503 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:02.673014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.565225 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:02.673646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.606635 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:02.673953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.640178 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 14:00:02.674264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.640282 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:00:02.674459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.674441 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:00:02.674591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.674579 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:00:02.674730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.674718 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:00:02.697008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.605210 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:02.718009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.699568 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:00:02.728014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.719143 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:02.774016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.766192 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j"] Nov 02 14:00:02.778580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.778165 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:00:02.866636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:02.839215 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:03.074008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.071189 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:03.074008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.071598 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:00:03.261138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.261106 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:00:03.286033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.270668 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" event={"ID":"1e82b09f-0e50-468c-a261-284d2407daa4","Type":"ContainerStarted","Data":"22359a3ecd8f23b89acdbe7dbb7e67a59d38e5951c900b0d5cf8d03a62e15cfa"} Nov 02 14:00:03.434023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.433905 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:00:03.459245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.444914 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 14:00:03.459245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.444977 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 14:00:03.554007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.550568 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:03.558099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.558056 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:03.636323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.636276 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:00:03.680290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.680244 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:03.706062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.680418 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:03.809009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.684042 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.006072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.697439 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.006072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.723209 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.006072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.798094 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:00:04.006072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:03.875166 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:04.006072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.003398 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.112009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.100706 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.146011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.133323 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh"] Nov 02 14:00:04.358017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.356169 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:04.358017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.356237 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.358498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.358378 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:04.358498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.358433 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.454191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.454143 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.460129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.460096 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 14:00:04.460384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.460356 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 14:00:04.535140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.535103 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:04.551269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.535383 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:04.636007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.631377 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" event={"ID":"db249407-60c6-4de9-8811-6e401a260bbf","Type":"ContainerStarted","Data":"8526b13670cb80c4376c8f74f1f74b7cc31164c5219b8bba3a609d256df71a13"} Nov 02 14:00:04.640078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.638467 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.688487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.688440 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:04.852050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.852000 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:00:04.940072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:04.912563 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw"] Nov 02 14:00:05.147072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.138445 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:00:05.178063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.174067 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 14:00:05.262067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.182439 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:05.327010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.182471 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:00:05.510009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.470298 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.586892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.476516 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:05.587416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.477057 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 14:00:05.631257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.631200 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 14:00:05.631755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.627932 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.632121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.632100 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.643252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.643126 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.643907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.643884 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmq2\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-kube-api-access-rnmq2\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.644172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.644153 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.644334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.644310 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.644495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.644466 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.644776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.644760 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.644971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.644954 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.645135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.529364 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 14:00:05.649007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.501650 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:05.649267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.649196 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:05.703049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.696062 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 14:00:05.703049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.696478 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Nov 02 14:00:05.703049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.701302 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:05.802285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.802240 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.806007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.802566 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.819008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.810144 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.820313 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.820357 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.820443 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmq2\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-kube-api-access-rnmq2\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.820561 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.820597 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.820661 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.821389 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.820193 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw"] Nov 02 14:00:05.823008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.821728 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.823567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.823347 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.841074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.837276 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:00:05.864354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.864318 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.869148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.867701 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.901783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.872872 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:05.948104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.874187 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:05.956009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.876076 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:05.981139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:05.981091 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:06.076750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.069360 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Nov 02 14:00:06.098371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.089969 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:06.099033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.092270 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:06.283417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.132281 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmq2\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-kube-api-access-rnmq2\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:06.283697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.157462 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:06.283909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.209540 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:06.284153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.284077 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:06.284323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.244494 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:06.399009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.396324 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:06.516204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.516140 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:06.550024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.526185 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:06.550024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.526247 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:06.594612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.594570 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:06.606034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.604543 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 14:00:06.606034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.604604 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 14:00:06.759531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.751387 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:06.759531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.751662 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:06.957641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:06.948626 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:07.092544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.092463 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:00:07.122149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.122106 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:07.132833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.124903 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:00:07.132833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.127480 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:00:07.139006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.137328 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:00:07.139184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.139026 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:00:07.149055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.139918 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:00:07.208003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.207597 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:00:07.218756 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:07.218724 5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c0304a_3d10_4b71_93f2_fe27f62b0ebc.slice\": RecentStats: unable to find data in memory cache]" Nov 02 14:00:07.459853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.459799 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 14:00:07.471853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.471781 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 14:00:07.496233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.496200 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw"] Nov 02 14:00:07.496543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.496519 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:07.496709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.496687 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:07.501767 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:00:07.501730 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c0304a_3d10_4b71_93f2_fe27f62b0ebc.slice/crio-fca301fdd117eda58f436af962c01a40739fb792c00a4ccfe10886273219be3b WatchSource:0}: Error finding container fca301fdd117eda58f436af962c01a40739fb792c00a4ccfe10886273219be3b: Status 404 returned error can't find the container with id fca301fdd117eda58f436af962c01a40739fb792c00a4ccfe10886273219be3b Nov 02 14:00:07.509569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.509532 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:07.509670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.509617 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:07.509670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.509658 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:07.941244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:07.938250 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:08.123007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:08.122827 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" event={"ID":"43c0304a-3d10-4b71-93f2-fe27f62b0ebc","Type":"ContainerStarted","Data":"c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999"} Nov 02 14:00:08.123007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:08.122898 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" event={"ID":"43c0304a-3d10-4b71-93f2-fe27f62b0ebc","Type":"ContainerStarted","Data":"fca301fdd117eda58f436af962c01a40739fb792c00a4ccfe10886273219be3b"} Nov 02 14:00:08.201093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:08.197343 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podStartSLOduration=4.197321573 podStartE2EDuration="4.197321573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:08.191729667 +0000 UTC m=+956.917351436" watchObservedRunningTime="2026-11-02 14:00:08.197321573 +0000 UTC m=+956.922943344" Nov 02 14:00:08.397864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:08.397832 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:08.399144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:08.399113 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 14:00:08.399266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:08.399167 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 14:00:08.497211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:08.497124 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:08.497211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:08.497188 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:09.425010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.413400 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 14:00:09.425010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.413466 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 14:00:09.505010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.503644 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:09.505010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.503706 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:09.514965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.514567 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:09.514965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.514629 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:09.620064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.620031 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 14:00:09.621289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.621267 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 14:00:09.621977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:09.621956 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 14:00:10.154722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:10.154617 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:10.398322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:10.397825 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 14:00:10.398322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:10.397893 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 14:00:10.496504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:10.496409 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 14:00:10.496504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:10.496476 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 14:00:11.399041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:11.398388 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 14:00:11.399041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:11.398448 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 14:00:12.325812 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:12.322746 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:12.401003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:12.400404 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 14:00:12.401003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:12.400479 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 14:00:12.564460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:12.564423 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:12.564745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:12.564717 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:13.203675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.203636 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 14:00:13.204037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.204013 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 14:00:13.204876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.204858 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 14:00:13.289064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.289026 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" event={"ID":"1e82b09f-0e50-468c-a261-284d2407daa4","Type":"ContainerStarted","Data":"d4ee39daa9a6907b246468f73fc941b94cfcd84e17924678f9d2fd3547421cce"} Nov 02 14:00:13.290729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.290702 5591 generic.go:356] "Generic (PLEG): container finished" podID="db249407-60c6-4de9-8811-6e401a260bbf" containerID="8880203c695a08253576516ac42d044493e7d2bd4c227dec6d17cab703c9a411" exitCode=60 Nov 02 14:00:13.290857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.290782 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" event={"ID":"db249407-60c6-4de9-8811-6e401a260bbf","Type":"ContainerDied","Data":"8880203c695a08253576516ac42d044493e7d2bd4c227dec6d17cab703c9a411"} Nov 02 14:00:13.290968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.290952 5591 scope.go:122] "RemoveContainer" containerID="8880203c695a08253576516ac42d044493e7d2bd4c227dec6d17cab703c9a411" Nov 02 14:00:13.397421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.397340 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 14:00:13.397421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.397386 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 14:00:13.408807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:13.408758 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" podStartSLOduration=4.416806698 podStartE2EDuration="13.408744845s" totalImagesPullingTime="8.991938147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:13.40628639 +0000 UTC m=+962.131908159" watchObservedRunningTime="2026-11-02 14:00:13.408744845 +0000 UTC m=+962.134366614" Nov 02 14:00:14.299739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:14.299704 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" event={"ID":"db249407-60c6-4de9-8811-6e401a260bbf","Type":"ContainerStarted","Data":"cf1bfbdca04b0c2bd772a1f2658d89ebba6588ebd4e3f0168d682ddf6aa76f50"} Nov 02 14:00:14.301390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:14.301361 5591 generic.go:356] "Generic (PLEG): container finished" podID="1e82b09f-0e50-468c-a261-284d2407daa4" containerID="d4ee39daa9a6907b246468f73fc941b94cfcd84e17924678f9d2fd3547421cce" exitCode=60 Nov 02 14:00:14.301526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:14.301442 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" event={"ID":"1e82b09f-0e50-468c-a261-284d2407daa4","Type":"ContainerDied","Data":"d4ee39daa9a6907b246468f73fc941b94cfcd84e17924678f9d2fd3547421cce"} Nov 02 14:00:14.301613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:14.301600 5591 scope.go:122] "RemoveContainer" containerID="d4ee39daa9a6907b246468f73fc941b94cfcd84e17924678f9d2fd3547421cce" Nov 02 14:00:14.352831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:14.352755 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" podStartSLOduration=6.335318275 podStartE2EDuration="14.352734382s" totalImagesPullingTime="8.017416107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:14.350490721 +0000 UTC m=+963.076112491" watchObservedRunningTime="2026-11-02 14:00:14.352734382 +0000 UTC m=+963.078356152" Nov 02 14:00:14.397207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:14.397173 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 14:00:14.397380 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:14.397230 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 14:00:14.543322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:14.543224 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:15.310371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:15.310332 5591 generic.go:356] "Generic (PLEG): container finished" podID="db249407-60c6-4de9-8811-6e401a260bbf" containerID="cf1bfbdca04b0c2bd772a1f2658d89ebba6588ebd4e3f0168d682ddf6aa76f50" exitCode=60 Nov 02 14:00:15.310591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:15.310413 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" event={"ID":"db249407-60c6-4de9-8811-6e401a260bbf","Type":"ContainerDied","Data":"cf1bfbdca04b0c2bd772a1f2658d89ebba6588ebd4e3f0168d682ddf6aa76f50"} Nov 02 14:00:15.310591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:15.310505 5591 scope.go:122] "RemoveContainer" containerID="8880203c695a08253576516ac42d044493e7d2bd4c227dec6d17cab703c9a411" Nov 02 14:00:15.310731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:15.310713 5591 scope.go:122] "RemoveContainer" containerID="cf1bfbdca04b0c2bd772a1f2658d89ebba6588ebd4e3f0168d682ddf6aa76f50" Nov 02 14:00:15.311130 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:15.311091 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh_odh-ai-gateway-infra(db249407-60c6-4de9-8811-6e401a260bbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" podUID="db249407-60c6-4de9-8811-6e401a260bbf" Nov 02 14:00:15.312658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:15.312634 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" event={"ID":"1e82b09f-0e50-468c-a261-284d2407daa4","Type":"ContainerStarted","Data":"564e32ec54eae6ede69cebb370de3d35325ae3e3917037c13df8817bd31bf9b7"} Nov 02 14:00:15.561898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:15.561802 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:15.563810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:15.562831 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:15.564283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:15.564258 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:00:16.319865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:16.319830 5591 scope.go:122] "RemoveContainer" containerID="cf1bfbdca04b0c2bd772a1f2658d89ebba6588ebd4e3f0168d682ddf6aa76f50" Nov 02 14:00:16.320238 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:16.320205 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh_odh-ai-gateway-infra(db249407-60c6-4de9-8811-6e401a260bbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" podUID="db249407-60c6-4de9-8811-6e401a260bbf" Nov 02 14:00:16.321700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:16.321674 5591 generic.go:356] "Generic (PLEG): container finished" podID="1e82b09f-0e50-468c-a261-284d2407daa4" containerID="564e32ec54eae6ede69cebb370de3d35325ae3e3917037c13df8817bd31bf9b7" exitCode=60 Nov 02 14:00:16.321820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:16.321727 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" event={"ID":"1e82b09f-0e50-468c-a261-284d2407daa4","Type":"ContainerDied","Data":"564e32ec54eae6ede69cebb370de3d35325ae3e3917037c13df8817bd31bf9b7"} Nov 02 14:00:16.321820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:16.321792 5591 scope.go:122] "RemoveContainer" containerID="d4ee39daa9a6907b246468f73fc941b94cfcd84e17924678f9d2fd3547421cce" Nov 02 14:00:16.322063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:16.322047 5591 scope.go:122] "RemoveContainer" containerID="564e32ec54eae6ede69cebb370de3d35325ae3e3917037c13df8817bd31bf9b7" Nov 02 14:00:16.322342 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:16.322320 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-66v9j_odh-ai-gateway-infra(1e82b09f-0e50-468c-a261-284d2407daa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" Nov 02 14:00:16.387120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:16.387084 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:17.329238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:17.329209 5591 scope.go:122] "RemoveContainer" containerID="564e32ec54eae6ede69cebb370de3d35325ae3e3917037c13df8817bd31bf9b7" Nov 02 14:00:17.329630 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:17.329464 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-66v9j_odh-ai-gateway-infra(1e82b09f-0e50-468c-a261-284d2407daa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" Nov 02 14:00:18.075506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:18.075436 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:19.868230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:19.868196 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:21.229140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:21.229099 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:21.471170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:21.471119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:22.783438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:22.783392 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:23.166541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:23.166219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:23.461411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:23.461327 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:00:24.076382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:24.075760 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:25.678038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:25.666512 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:25.680125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:25.680072 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:25.713145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:25.713096 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:25.946185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:25.946080 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:25.953204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:25.953157 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:26.015009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.014297 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:26.019007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.015716 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:26.104964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.104914 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:26.226243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.226123 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:26.478266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.478128 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:00:26.667116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.667078 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:26.684141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.684086 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:26.685486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.685457 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:00:26.939481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.939426 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Nov 02 14:00:27.011014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:26.989415 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:27.053478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.019233 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:00:27.109009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.019364 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:27.117010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.058256 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:00:27.127144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.127106 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:00:27.158209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.158185 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:00:27.158748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.121092 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:27.159536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.080504 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:27.159745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.159728 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:00:27.242452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.242345 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:27.243876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.243850 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:00:27.514061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.509927 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:00:27.680701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.680667 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:00:27.690024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.687864 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:27.694620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.694595 5591 scope.go:122] "RemoveContainer" containerID="564e32ec54eae6ede69cebb370de3d35325ae3e3917037c13df8817bd31bf9b7" Nov 02 14:00:27.717184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.703616 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:27.737597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:27.735721 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:28.063022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.054180 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:00:28.095658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.091097 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:00:28.111137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.111093 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:28.165264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.165185 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 14:00:28.255178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.252136 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 14:00:28.518008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.482490 5591 scope.go:122] "RemoveContainer" containerID="cf1bfbdca04b0c2bd772a1f2658d89ebba6588ebd4e3f0168d682ddf6aa76f50" Nov 02 14:00:28.636447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.636394 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:28.985447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.688265 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:00:28.997014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.758965 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:29.006187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.792238 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:29.006419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:28.955086 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:29.006419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.006300 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:29.013016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.006867 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:00:29.099020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.098795 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:29.102161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.102116 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:29.128422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.128380 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:00:29.257069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.255072 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 14:00:29.257069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.256172 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:29.282142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.282094 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:29.282420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.282388 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:29.366017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.360003 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:29.674021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.673501 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx"] Nov 02 14:00:29.690805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:29.686774 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:30.229043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.214362 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:30.276042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.265536 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:30.278250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.270916 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:00:30.396939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.396888 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:00:30.400525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.400488 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 14:00:30.662236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.432363 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:30.662499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.456137 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-certs\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:30.662668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.547698 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d776122a-openshift-default-dockercfg-v92ck\"" Nov 02 14:00:30.744009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.728592 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:00:30.770188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.752356 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:30.903142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.903078 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:30.964694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.664297 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6z84\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-kube-api-access-b6z84\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:30.980357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.972185 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-socket\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:30.980631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.980606 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-envoy\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.009009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.982451 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-credential-socket\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.059016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.058242 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istiod-ca-cert\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.059016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.058419 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-data\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.059016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.058495 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-podinfo\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.059016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.058583 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-token\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.062644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.059759 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:31.118164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:30.979769 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx"] Nov 02 14:00:31.152567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.152538 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:00:31.152827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.152810 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:00:31.159604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.159565 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-data\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.159889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.159865 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-podinfo\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.160230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.160210 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-token\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.160414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.160387 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-certs\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.160558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.160540 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6z84\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-kube-api-access-b6z84\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.160741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.160725 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-socket\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.160887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.160868 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-envoy\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.161146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.161120 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-credential-socket\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.161293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.161276 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istiod-ca-cert\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.162398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.162377 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istiod-ca-cert\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.163014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.162968 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-data\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.176692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.176192 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-certs\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.178218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.177684 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-socket\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.178218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.177951 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-credential-socket\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.204394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.204351 5591 generic.go:356] "Generic (PLEG): container finished" podID="1e82b09f-0e50-468c-a261-284d2407daa4" containerID="de179bd235f8fe385c7ac7826d42b7d72287cfbddbc9ef1bc7f8b98068bdd765" exitCode=60 Nov 02 14:00:31.205079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.205042 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" event={"ID":"1e82b09f-0e50-468c-a261-284d2407daa4","Type":"ContainerDied","Data":"de179bd235f8fe385c7ac7826d42b7d72287cfbddbc9ef1bc7f8b98068bdd765"} Nov 02 14:00:31.206210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.206182 5591 scope.go:122] "RemoveContainer" containerID="564e32ec54eae6ede69cebb370de3d35325ae3e3917037c13df8817bd31bf9b7" Nov 02 14:00:31.216259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.216189 5591 scope.go:122] "RemoveContainer" containerID="de179bd235f8fe385c7ac7826d42b7d72287cfbddbc9ef1bc7f8b98068bdd765" Nov 02 14:00:31.237011 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:31.216803 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-66v9j_odh-ai-gateway-infra(1e82b09f-0e50-468c-a261-284d2407daa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" Nov 02 14:00:31.262008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.250293 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:00:31.262008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.251224 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-podinfo\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.386240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.310046 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:00:31.386240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.385134 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:00:31.406007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.404972 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:31.488795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.488760 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-envoy\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.489555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.489526 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6z84\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-kube-api-access-b6z84\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.523556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.523522 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-token\") pod \"e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.615507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.589053 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:31.615507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.594670 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:00:31.615507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.594854 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:00:31.625745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.625716 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:00:31.707894 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.707865 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:00:31.793729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:31.780901 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:31.856281 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:31.856240 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:00:32.278170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.278140 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx"] Nov 02 14:00:32.278508 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:00:32.278142 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1e13a3_e1c5_481a_b6da_ec915ab7a6f5.slice/crio-053f58d326d46901037375ce0dc2dcde3ad8941dfbdf42136e09d9aa9c807540 WatchSource:0}: Error finding container 053f58d326d46901037375ce0dc2dcde3ad8941dfbdf42136e09d9aa9c807540: Status 404 returned error can't find the container with id 053f58d326d46901037375ce0dc2dcde3ad8941dfbdf42136e09d9aa9c807540 Nov 02 14:00:32.281010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.280959 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:32.281092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.281058 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:32.281128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.281097 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:32.393656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.393619 5591 generic.go:356] "Generic (PLEG): container finished" podID="db249407-60c6-4de9-8811-6e401a260bbf" containerID="54fc2e87d16cf70ca9bb07593fa0e480d37fd886dbef1fc92129ef7104741cbf" exitCode=60 Nov 02 14:00:32.393874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.393709 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" event={"ID":"db249407-60c6-4de9-8811-6e401a260bbf","Type":"ContainerDied","Data":"54fc2e87d16cf70ca9bb07593fa0e480d37fd886dbef1fc92129ef7104741cbf"} Nov 02 14:00:32.393874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.393807 5591 scope.go:122] "RemoveContainer" containerID="cf1bfbdca04b0c2bd772a1f2658d89ebba6588ebd4e3f0168d682ddf6aa76f50" Nov 02 14:00:32.394318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.394295 5591 scope.go:122] "RemoveContainer" containerID="54fc2e87d16cf70ca9bb07593fa0e480d37fd886dbef1fc92129ef7104741cbf" Nov 02 14:00:32.394677 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:00:32.394650 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh_odh-ai-gateway-infra(db249407-60c6-4de9-8811-6e401a260bbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" podUID="db249407-60c6-4de9-8811-6e401a260bbf" Nov 02 14:00:32.396159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.396069 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" event={"ID":"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5","Type":"ContainerStarted","Data":"551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92"} Nov 02 14:00:32.396159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.396120 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" event={"ID":"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5","Type":"ContainerStarted","Data":"053f58d326d46901037375ce0dc2dcde3ad8941dfbdf42136e09d9aa9c807540"} Nov 02 14:00:32.451907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.451746 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j"] Nov 02 14:00:32.494000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.493308 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podStartSLOduration=3.493294528 podStartE2EDuration="3.493294528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:32.490538499 +0000 UTC m=+981.216160269" watchObservedRunningTime="2026-11-02 14:00:32.493294528 +0000 UTC m=+981.218916297" Nov 02 14:00:32.704007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.702892 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:32.725307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.725274 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e82b09f-0e50-468c-a261-284d2407daa4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e82b09f-0e50-468c-a261-284d2407daa4-maas-api-service-ca\") pod \"1e82b09f-0e50-468c-a261-284d2407daa4\" (UID: \"1e82b09f-0e50-468c-a261-284d2407daa4\") " Nov 02 14:00:32.725518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.725331 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e82b09f-0e50-468c-a261-284d2407daa4-kube-api-access-99mm5\" (UniqueName: \"kubernetes.io/projected/1e82b09f-0e50-468c-a261-284d2407daa4-kube-api-access-99mm5\") pod \"1e82b09f-0e50-468c-a261-284d2407daa4\" (UID: \"1e82b09f-0e50-468c-a261-284d2407daa4\") " Nov 02 14:00:32.725713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.725689 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e82b09f-0e50-468c-a261-284d2407daa4-maas-api-service-ca" pod "1e82b09f-0e50-468c-a261-284d2407daa4" (UID: "1e82b09f-0e50-468c-a261-284d2407daa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:00:32.726110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.726089 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e82b09f-0e50-468c-a261-284d2407daa4-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:00:32.728306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.728270 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e82b09f-0e50-468c-a261-284d2407daa4-kube-api-access-99mm5" pod "1e82b09f-0e50-468c-a261-284d2407daa4" (UID: "1e82b09f-0e50-468c-a261-284d2407daa4"). InnerVolumeSpecName "kube-api-access-99mm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:00:32.781741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.781709 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:32.827727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:32.827687 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99mm5\" (UniqueName: \"kubernetes.io/projected/1e82b09f-0e50-468c-a261-284d2407daa4-kube-api-access-99mm5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:00:33.119706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.119669 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:33.404822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.404794 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" Nov 02 14:00:33.405355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.404834 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j" event={"ID":"1e82b09f-0e50-468c-a261-284d2407daa4","Type":"ContainerDied","Data":"22359a3ecd8f23b89acdbe7dbb7e67a59d38e5951c900b0d5cf8d03a62e15cfa"} Nov 02 14:00:33.405355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.404906 5591 scope.go:122] "RemoveContainer" containerID="de179bd235f8fe385c7ac7826d42b7d72287cfbddbc9ef1bc7f8b98068bdd765" Nov 02 14:00:33.448519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.448486 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh"] Nov 02 14:00:33.462812 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.462786 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j"] Nov 02 14:00:33.474715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.474684 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-66v9j"] Nov 02 14:00:33.605595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.605565 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:33.638030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.637945 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db249407-60c6-4de9-8811-6e401a260bbf-kube-api-access-zzcfz\" (UniqueName: \"kubernetes.io/projected/db249407-60c6-4de9-8811-6e401a260bbf-kube-api-access-zzcfz\") pod \"db249407-60c6-4de9-8811-6e401a260bbf\" (UID: \"db249407-60c6-4de9-8811-6e401a260bbf\") " Nov 02 14:00:33.638030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.637999 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db249407-60c6-4de9-8811-6e401a260bbf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db249407-60c6-4de9-8811-6e401a260bbf-maas-api-service-ca\") pod \"db249407-60c6-4de9-8811-6e401a260bbf\" (UID: \"db249407-60c6-4de9-8811-6e401a260bbf\") " Nov 02 14:00:33.638565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.638514 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db249407-60c6-4de9-8811-6e401a260bbf-maas-api-service-ca" pod "db249407-60c6-4de9-8811-6e401a260bbf" (UID: "db249407-60c6-4de9-8811-6e401a260bbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:00:33.638801 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.638777 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db249407-60c6-4de9-8811-6e401a260bbf-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:00:33.640305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.640272 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db249407-60c6-4de9-8811-6e401a260bbf-kube-api-access-zzcfz" pod "db249407-60c6-4de9-8811-6e401a260bbf" (UID: "db249407-60c6-4de9-8811-6e401a260bbf"). InnerVolumeSpecName "kube-api-access-zzcfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:00:33.740782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.740702 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzcfz\" (UniqueName: \"kubernetes.io/projected/db249407-60c6-4de9-8811-6e401a260bbf-kube-api-access-zzcfz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:00:33.788321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:33.788288 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:34.414205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:34.414176 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" Nov 02 14:00:34.414700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:34.414209 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh" event={"ID":"db249407-60c6-4de9-8811-6e401a260bbf","Type":"ContainerDied","Data":"8526b13670cb80c4376c8f74f1f74b7cc31164c5219b8bba3a609d256df71a13"} Nov 02 14:00:34.414700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:34.414270 5591 scope.go:122] "RemoveContainer" containerID="54fc2e87d16cf70ca9bb07593fa0e480d37fd886dbef1fc92129ef7104741cbf" Nov 02 14:00:34.416014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:34.415964 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:34.417221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:34.417201 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:00:34.479996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:34.479954 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh"] Nov 02 14:00:34.491967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:34.491939 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893800-txtjh"] Nov 02 14:00:35.132370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:35.132336 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:35.386191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:35.381582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:35.434257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:35.434222 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" path="/var/lib/kubelet/pods/1e82b09f-0e50-468c-a261-284d2407daa4/volumes" Nov 02 14:00:35.435150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:35.435123 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db249407-60c6-4de9-8811-6e401a260bbf" path="/var/lib/kubelet/pods/db249407-60c6-4de9-8811-6e401a260bbf/volumes" Nov 02 14:00:36.881631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:36.881603 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:37.081243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.081213 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:37.109172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109115 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8"] Nov 02 14:00:37.109814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109784 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:37.109814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109806 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:37.109814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109819 5591 cpu_manager.go:414] "containerMap: removing container" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:37.109814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109824 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109835 5591 cpu_manager.go:414] "containerMap: removing container" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109841 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109856 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109861 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.109973 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.110002 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.110011 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.110019 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:37.110249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.110029 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:37.123046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.116578 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:37.152007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.142342 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8"] Nov 02 14:00:37.162011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.154608 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d776122a-dockercfg-cjq2d\"" Nov 02 14:00:37.162011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.155250 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d776122a\"" Nov 02 14:00:37.209009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.208715 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8\" (UID: \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:37.209009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.208774 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28x5j\" (UniqueName: \"kubernetes.io/projected/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-kube-api-access-28x5j\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8\" (UID: \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:37.360006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.357491 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8\" (UID: \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:37.360006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.357561 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28x5j\" (UniqueName: \"kubernetes.io/projected/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-kube-api-access-28x5j\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8\" (UID: \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:37.360006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.359080 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-plugins-config-volume\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8\" (UID: \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:37.477056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.474756 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28x5j\" (UniqueName: \"kubernetes.io/projected/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-kube-api-access-28x5j\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8\" (UID: \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:37.491206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.488912 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:37.517008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.513104 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:37.826997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:37.826958 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8"] Nov 02 14:00:37.834183 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:00:37.834147 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a0ac73_db13_4d3c_a0a7_239c4b2d722c.slice/crio-91051cee8b8cac501d911717c5187f3250ca71005af3b081a1aa2eed54fa0a72 WatchSource:0}: Error finding container 91051cee8b8cac501d911717c5187f3250ca71005af3b081a1aa2eed54fa0a72: Status 404 returned error can't find the container with id 91051cee8b8cac501d911717c5187f3250ca71005af3b081a1aa2eed54fa0a72 Nov 02 14:00:38.518271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:38.518238 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" event={"ID":"21a0ac73-db13-4d3c-a0a7-239c4b2d722c","Type":"ContainerStarted","Data":"dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85"} Nov 02 14:00:38.518673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:38.518287 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" event={"ID":"21a0ac73-db13-4d3c-a0a7-239c4b2d722c","Type":"ContainerStarted","Data":"91051cee8b8cac501d911717c5187f3250ca71005af3b081a1aa2eed54fa0a72"} Nov 02 14:00:38.518673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:38.518339 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:38.567897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:38.567846 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" podStartSLOduration=1.567831676 podStartE2EDuration="1.567831676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:38.567723571 +0000 UTC m=+987.293345341" watchObservedRunningTime="2026-11-02 14:00:38.567831676 +0000 UTC m=+987.293453445" Nov 02 14:00:39.470010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:39.469247 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:41.189008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:41.188650 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:42.977014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:42.976150 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:44.110048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.110016 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8"] Nov 02 14:00:44.110574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.110561 5591 cpu_manager.go:414] "containerMap: removing container" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:44.110621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.110577 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db249407-60c6-4de9-8811-6e401a260bbf" containerName="cleanup" Nov 02 14:00:44.110621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.110605 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:44.110621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.110612 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:44.110733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.110724 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e82b09f-0e50-468c-a261-284d2407daa4" containerName="cleanup" Nov 02 14:00:44.115195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.115176 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.142564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.142531 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8"] Nov 02 14:00:44.221188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221152 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.221188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221190 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.221454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221225 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.221454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221386 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.221454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221424 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhb2r\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-kube-api-access-zhb2r\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.221593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221463 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.221669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221646 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ef2f4d-98a0-41d0-8a68-75a21356effb-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.221728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221713 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.221780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.221768 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323243 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323283 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323320 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323343 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323358 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhb2r\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-kube-api-access-zhb2r\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323381 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323497 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ef2f4d-98a0-41d0-8a68-75a21356effb-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323554 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323579 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323749 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.323800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323771 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.324035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323907 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.324035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.323931 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.324335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.324309 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ef2f4d-98a0-41d0-8a68-75a21356effb-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.325886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.325867 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.326113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.326097 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.347754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.347722 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.347868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.347843 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhb2r\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-kube-api-access-zhb2r\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.433395 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.433364 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:44.761770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.761604 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:44.776324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.773318 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8"] Nov 02 14:00:44.789293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.789248 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:44.789538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.789514 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:44.789676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:44.789660 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:00:45.570597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:45.570558 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" event={"ID":"47ef2f4d-98a0-41d0-8a68-75a21356effb","Type":"ContainerStarted","Data":"14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb"} Nov 02 14:00:45.571057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:45.570616 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" event={"ID":"47ef2f4d-98a0-41d0-8a68-75a21356effb","Type":"ContainerStarted","Data":"4ff255ca697a932fca76b5c91299d70177b505fcd917bd8ffe701ce983d1eb86"} Nov 02 14:00:45.640864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:45.640791 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podStartSLOduration=1.640770178 podStartE2EDuration="1.640770178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:45.638234959 +0000 UTC m=+994.363856729" watchObservedRunningTime="2026-11-02 14:00:45.640770178 +0000 UTC m=+994.366391949" Nov 02 14:00:46.451417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:46.451380 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:46.457018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:46.456943 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:46.457210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:46.457144 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:46.457847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:46.457381 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:46.658557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:46.658518 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:47.438013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:47.436376 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:47.438013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:47.436473 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:48.455007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:48.444804 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:48.455007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:48.444872 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:48.628012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:48.627932 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:49.450636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:49.450604 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:49.450943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:49.450911 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:49.575011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:49.565226 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:00:50.442450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:50.442410 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:50.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:50.444523 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:50.800097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:50.787089 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:50.862568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:50.862164 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:51.147068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.146723 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:51.152007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.148256 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:00:51.152007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.146293 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:00:51.260129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.260078 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:51.315957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.315909 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:51.447066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.442469 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:51.447066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.442533 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:51.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.789718 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:00:51.790309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.790286 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:00:51.792184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.792136 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:51.863109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.863049 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:51.864271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:51.864237 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:00:52.010254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.010208 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:52.165013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.150084 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:52.165013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.150865 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:00:52.165013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.152425 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:52.165013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.152761 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:00:52.165013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.153865 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:52.174017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.165775 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:00:52.267693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.265008 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 14:00:52.267693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.265490 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:00:52.303379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.303326 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:52.303668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.303616 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:52.323140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.323067 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:52.324678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.324652 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:00:52.464118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.457595 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:52.464118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.457656 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:52.810245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.806787 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:52.833965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.833641 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 02 14:00:52.866055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:52.866011 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:53.054132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.054091 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:53.054403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.054367 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:53.064061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.056229 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:53.077757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.077720 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:00:53.155114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.151144 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:00:53.155304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.155183 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:00:53.221022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.158252 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:00:53.330090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.326384 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:53.360026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.355944 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 14:00:53.563007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.558126 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:53.563007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.558216 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:53.631069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.627338 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:53.761035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.713506 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:53.762486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.762440 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:53.879974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.810611 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:00:53.916068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:53.904079 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:54.079275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.079224 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:54.169063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.168147 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:54.187034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.171591 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:54.303120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.303065 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:00:54.356009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.333953 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:54.358388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.358349 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:54.453204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.453109 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:54.453476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.453443 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:54.914031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.913892 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:00:54.977371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:54.977329 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.110655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.110603 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.194318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.194215 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.196536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.194143 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:00:55.283207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.283155 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cjzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:55.283508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.283449 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.334007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.322319 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.336149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.336108 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-nvjl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:55.336386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.336354 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.356466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.356424 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 14:00:55.360467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.360430 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.442307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.442272 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:55.442948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.442890 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:55.696007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.672205 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.696318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.691323 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:55.696518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.696482 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.764677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.764622 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:55.812877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.769025 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:00:55.836006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.832368 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:00:55.920005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.918175 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:55.978806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:55.978712 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:56.093387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.093351 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:00:56.137047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.135595 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:00:56.353105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.349150 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:00:56.364146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.364095 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:56.481619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.481590 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:00:56.488247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.488222 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:00:56.488494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.488475 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:00:56.491206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.491173 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:56.491315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.491229 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:56.513042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.494595 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:00:56.565267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:56.565236 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:00:57.441364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:57.441325 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:00:57.442109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:57.442075 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:00:58.132531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:58.132492 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:59.396763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:59.396732 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:59.397342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:59.397312 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:00:59.398114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:00:59.398088 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:01:00.157702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:00.157657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:01.074090 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:01:01.074053 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:01:02.049043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:02.048962 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:03.806826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:03.806775 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:05.563586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:05.563539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:07.361667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:07.361629 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:09.238369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:09.238338 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:09.407946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:09.407907 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:10.351392 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:10.351255 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:11.336007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:11.332169 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:11.798007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:11.790203 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:01:13.358220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:13.358179 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:13.728009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:13.727090 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:13.817021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:13.811069 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:14.123218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:14.123101 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:14.203205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:14.203144 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:14.306190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:14.306130 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:14.334010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:14.332092 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 14:01:14.375221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:14.375114 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:14.927009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:14.926407 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 14:01:15.109008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.108665 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:15.123009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.109404 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:01:15.162031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.161020 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:15.162031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.161093 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:15.162031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.161416 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:01:15.462115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.345166 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:15.462115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.345593 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:01:15.462115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.345936 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:01:15.462115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.346016 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:01:15.462115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.433708 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:15.462115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.434201 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:01:15.729125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.728568 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Nov 02 14:01:15.827632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:15.827583 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:16.103067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.095346 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 14:01:16.198466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.186090 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:16.227043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.186414 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 02 14:01:16.227822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.189661 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:16.227822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.189709 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:01:16.228340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.212410 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:16.259550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.246406 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk"] Nov 02 14:01:16.259550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.251583 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:01:16.259550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.251639 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:01:16.259550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.251676 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:01:16.259550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.252611 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.327301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.291319 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d776122a\"/\"kube-root-ca.crt\"" Nov 02 14:01:16.327301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.299784 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d776122a\"/\"default-dockercfg-2l8c8\"" Nov 02 14:01:16.327301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.300377 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d776122a\"/\"e2e-lifecycle-model-d776122a-kserve-self-signed-certs\"" Nov 02 14:01:16.344571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.344537 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tmp-dir\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.345434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.345400 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:16.346050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.346023 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:16.349574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.345209 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-model-cache\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.364176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.364086 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-home\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.414908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.413299 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-dshm\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.414908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.413364 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f5dz\" (UniqueName: \"kubernetes.io/projected/99c3140d-c1e2-4d56-9d7c-6090de669c2a-kube-api-access-5f5dz\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.414908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.413542 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tls-certs\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.414908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.382207 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d776122a\"/\"openshift-service-ca.crt\"" Nov 02 14:01:16.447010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.443577 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:16.497009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.478634 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk"] Nov 02 14:01:16.634896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.631105 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-home\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.634896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.631280 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-dshm\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.634896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.631322 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f5dz\" (UniqueName: \"kubernetes.io/projected/99c3140d-c1e2-4d56-9d7c-6090de669c2a-kube-api-access-5f5dz\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.634896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.631419 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tls-certs\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.634896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.631484 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tmp-dir\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.634896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.631514 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-model-cache\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.787009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.631969 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-model-cache\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.903063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.640367 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:16.903063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.640951 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-home\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.903063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.658552 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tmp-dir\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.903063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.902723 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tls-certs\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.903483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.903446 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f5dz\" (UniqueName: \"kubernetes.io/projected/99c3140d-c1e2-4d56-9d7c-6090de669c2a-kube-api-access-5f5dz\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.903719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.903696 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-dshm\") pod \"e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.935106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.935065 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:16.938712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.938666 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 14:01:16.947492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:16.947441 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Nov 02 14:01:17.195061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.193023 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:17.227020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.210265 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:01:17.227020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.218246 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:17.270591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.237199 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:17.270591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.238739 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:17.358010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.346172 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:17.358010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.348149 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:17.517060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.515043 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:17.578065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.578032 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:01:17.857236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.854321 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:01:17.922308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.913494 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:01:17.978149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.978104 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Nov 02 14:01:17.994881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:17.980483 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 14:01:18.250837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.240276 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:01:18.323015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.322642 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk"] Nov 02 14:01:18.413016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.410861 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:18.413759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.413726 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:01:18.521318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.521237 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" event={"ID":"99c3140d-c1e2-4d56-9d7c-6090de669c2a","Type":"ContainerStarted","Data":"6c78326b95b062a011b0f9f844ac7cd5f87f28d57530d44abed1afe0cf8af8ba"} Nov 02 14:01:18.521842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.515966 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:18.701500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.700334 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:01:18.897329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.891275 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 14:01:18.950277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:18.949062 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:19.047197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.047147 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:19.109006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.108877 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:19.306216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.306116 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:19.411160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.411116 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:19.447415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.445534 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:19.518946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.518910 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:01:19.519140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.519025 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:01:19.519140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.519069 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:01:19.530744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.530719 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:01:19.559643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.559543 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" event={"ID":"99c3140d-c1e2-4d56-9d7c-6090de669c2a","Type":"ContainerStarted","Data":"933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816"} Nov 02 14:01:19.716560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.716527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:19.778563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.778500 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" podStartSLOduration=4.778479724 podStartE2EDuration="4.778479724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:19.772436693 +0000 UTC m=+1028.498058473" watchObservedRunningTime="2026-11-02 14:01:19.778479724 +0000 UTC m=+1028.504101498" Nov 02 14:01:19.975105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:19.975072 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:20.002024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:20.001959 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:20.884003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:20.873558 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:21.356068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:21.351902 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:21.917042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:21.916967 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:22.572063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.572015 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s"] Nov 02 14:01:22.592274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.592242 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s"] Nov 02 14:01:22.602035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.592562 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.651044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.650998 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9eb67b-openshift-default-dockercfg-q2jfv\"" Nov 02 14:01:22.678806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.671192 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:22.797586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.797546 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-envoy\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.798070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.798044 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-token\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.798379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.798350 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-podinfo\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.798780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.798759 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-socket\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.799060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.799040 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-credential-socket\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.799325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.799306 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istiod-ca-cert\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.799518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.799501 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfbmb\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-kube-api-access-bfbmb\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.799751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.799735 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-certs\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.800056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.800028 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-data\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.884902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.884812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:22.906949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.906833 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-token\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.906949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.906931 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-podinfo\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.907229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.906973 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-socket\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.907229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.907103 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-credential-socket\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.907229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.907149 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istiod-ca-cert\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.907229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.907174 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfbmb\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-kube-api-access-bfbmb\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.907897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.907270 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-certs\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.907897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.907436 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-data\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.907897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.907508 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-envoy\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.939459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.919181 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-credential-socket\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.939459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.921054 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istiod-ca-cert\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.939459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.921318 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-certs\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.939459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.921565 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-socket\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.939459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.921850 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-data\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.965013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.960591 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-token\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.996643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.979474 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfbmb\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-kube-api-access-bfbmb\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.996643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.993478 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-envoy\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:22.996643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:22.993719 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-podinfo\") pod \"e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:23.032140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.021298 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:23.053948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.053876 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:23.415792 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:01:23.415757 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7de0a8a_dd54_41d3_9444_d8a6e6667f2c.slice/crio-8263908f2bd1d58cc9f253a0813379e8fbe3cf716282b72148c6a7c5b3508f4c WatchSource:0}: Error finding container 8263908f2bd1d58cc9f253a0813379e8fbe3cf716282b72148c6a7c5b3508f4c: Status 404 returned error can't find the container with id 8263908f2bd1d58cc9f253a0813379e8fbe3cf716282b72148c6a7c5b3508f4c Nov 02 14:01:23.416449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.416422 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s"] Nov 02 14:01:23.419178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.419134 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:01:23.419299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.419224 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:01:23.419299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.419264 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:01:23.702954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.702862 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" event={"ID":"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c","Type":"ContainerStarted","Data":"ba4db4d3bc4e2081554ae3daeef27910d4de9e203d5ccf4ef79a843f04c382e1"} Nov 02 14:01:23.702954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.702919 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" event={"ID":"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c","Type":"ContainerStarted","Data":"8263908f2bd1d58cc9f253a0813379e8fbe3cf716282b72148c6a7c5b3508f4c"} Nov 02 14:01:23.755975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:23.755921 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podStartSLOduration=1.755905409 podStartE2EDuration="1.755905409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:23.752911686 +0000 UTC m=+1032.478533457" watchObservedRunningTime="2026-11-02 14:01:23.755905409 +0000 UTC m=+1032.481527178" Nov 02 14:01:24.025230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:24.025132 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:24.031105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:24.031071 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:24.126272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:24.126239 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:24.290029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:24.289907 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:24.709508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:24.709476 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:24.710415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:24.710391 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:01:24.870551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:24.870518 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:25.815183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.815152 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:25.882628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.882598 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc"] Nov 02 14:01:25.887379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.887357 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:25.912504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.912473 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc"] Nov 02 14:01:25.927804 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.922244 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4c6741-dockercfg-f98kt\"" Nov 02 14:01:25.927804 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.922258 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4c6741\"" Nov 02 14:01:25.946788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.946756 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk"] Nov 02 14:01:25.951730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.951705 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:25.963261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.963231 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvtt\" (UniqueName: \"kubernetes.io/projected/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-kube-api-access-qvvtt\") pod \"payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc\" (UID: \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:25.965176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.963618 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc\" (UID: \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:25.968506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:25.968479 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk"] Nov 02 14:01:26.068082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.067300 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc\" (UID: \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:26.068082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.067375 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb583eb-b763-42f5-bc28-e651ae7e8d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk\" (UID: \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:26.068082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.067432 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxfr\" (UniqueName: \"kubernetes.io/projected/6cb583eb-b763-42f5-bc28-e651ae7e8d23-kube-api-access-4zxfr\") pod \"payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk\" (UID: \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:26.068082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.067512 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvtt\" (UniqueName: \"kubernetes.io/projected/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-kube-api-access-qvvtt\") pod \"payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc\" (UID: \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:26.152890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.152851 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:26.153922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.153897 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc\" (UID: \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:26.161776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.161435 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvtt\" (UniqueName: \"kubernetes.io/projected/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-kube-api-access-qvvtt\") pod \"payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc\" (UID: \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:26.186679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.178915 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb583eb-b763-42f5-bc28-e651ae7e8d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk\" (UID: \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:26.186679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.180085 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb583eb-b763-42f5-bc28-e651ae7e8d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk\" (UID: \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:26.186679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.183876 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxfr\" (UniqueName: \"kubernetes.io/projected/6cb583eb-b763-42f5-bc28-e651ae7e8d23-kube-api-access-4zxfr\") pod \"payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk\" (UID: \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:26.225046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.210799 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:26.259036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.242974 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxfr\" (UniqueName: \"kubernetes.io/projected/6cb583eb-b763-42f5-bc28-e651ae7e8d23-kube-api-access-4zxfr\") pod \"payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk\" (UID: \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:26.335475 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.335378 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:26.346991 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.346944 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:26.609024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.608976 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc"] Nov 02 14:01:26.609631 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:01:26.609591 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5caf66_7ef3_4f97_8c22_f2c0fef0e170.slice/crio-ba3e16583108524c6870dbf2d191d7a0eb157257e7c8ce8ff09c4d229d4550aa WatchSource:0}: Error finding container ba3e16583108524c6870dbf2d191d7a0eb157257e7c8ce8ff09c4d229d4550aa: Status 404 returned error can't find the container with id ba3e16583108524c6870dbf2d191d7a0eb157257e7c8ce8ff09c4d229d4550aa Nov 02 14:01:26.680873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.680843 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk"] Nov 02 14:01:26.681585 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:01:26.681560 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb583eb_b763_42f5_bc28_e651ae7e8d23.slice/crio-32f35bc2685c045178b190e38d4cc7d688a89b6417903ab1167c35eb9698bd60 WatchSource:0}: Error finding container 32f35bc2685c045178b190e38d4cc7d688a89b6417903ab1167c35eb9698bd60: Status 404 returned error can't find the container with id 32f35bc2685c045178b190e38d4cc7d688a89b6417903ab1167c35eb9698bd60 Nov 02 14:01:26.723717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.723681 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" event={"ID":"6cb583eb-b763-42f5-bc28-e651ae7e8d23","Type":"ContainerStarted","Data":"32f35bc2685c045178b190e38d4cc7d688a89b6417903ab1167c35eb9698bd60"} Nov 02 14:01:26.725665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.725632 5591 generic.go:356] "Generic (PLEG): container finished" podID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerID="551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92" exitCode=0 Nov 02 14:01:26.725799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.725668 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" event={"ID":"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5","Type":"ContainerDied","Data":"551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92"} Nov 02 14:01:26.726199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.726173 5591 scope.go:122] "RemoveContainer" containerID="551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92" Nov 02 14:01:26.728011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.727946 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" event={"ID":"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170","Type":"ContainerStarted","Data":"c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722"} Nov 02 14:01:26.728125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.728017 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" event={"ID":"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170","Type":"ContainerStarted","Data":"ba3e16583108524c6870dbf2d191d7a0eb157257e7c8ce8ff09c4d229d4550aa"} Nov 02 14:01:26.728332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.728310 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:26.728408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.728362 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:01:26.728448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.728427 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:01:26.728520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.728501 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:01:26.825587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.825527 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" podStartSLOduration=1.825511438 podStartE2EDuration="1.825511438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:26.823109416 +0000 UTC m=+1035.548731186" watchObservedRunningTime="2026-11-02 14:01:26.825511438 +0000 UTC m=+1035.551133211" Nov 02 14:01:26.966006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:26.965107 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:27.072828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.072796 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:27.100257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.100215 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:27.762605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.762561 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" event={"ID":"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5","Type":"ContainerStarted","Data":"0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f"} Nov 02 14:01:27.764725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.764681 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" event={"ID":"6cb583eb-b763-42f5-bc28-e651ae7e8d23","Type":"ContainerStarted","Data":"ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946"} Nov 02 14:01:27.776487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.776462 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:01:27.781808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.781784 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:01:27.781971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.781951 5591 patch_prober.go:29] interesting pod/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 14:01:27.782047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.782022 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 14:01:27.858280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.858234 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:27.910049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:27.909975 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" podStartSLOduration=2.909961819 podStartE2EDuration="2.909961819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:27.908887304 +0000 UTC m=+1036.634509073" watchObservedRunningTime="2026-11-02 14:01:27.909961819 +0000 UTC m=+1036.635583589" Nov 02 14:01:28.064150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:28.064032 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:28.776673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:28.776637 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:28.781956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:28.781925 5591 patch_prober.go:29] interesting pod/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 14:01:28.782135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:28.781995 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 14:01:28.947010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:28.943065 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:29.781585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:29.781557 5591 patch_prober.go:29] interesting pod/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 14:01:29.781767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:29.781605 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 14:01:30.764004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:30.758843 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:30.786009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:30.785420 5591 patch_prober.go:29] interesting pod/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 14:01:30.786009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:30.785504 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 14:01:31.082578 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:01:31.082489 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:01:31.783006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:31.781815 5591 patch_prober.go:29] interesting pod/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 14:01:31.783006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:31.781875 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 14:01:32.665005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:32.664510 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:32.786010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:32.783556 5591 patch_prober.go:29] interesting pod/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 14:01:32.786010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:32.783622 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 14:01:33.790516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:33.790473 5591 patch_prober.go:29] interesting pod/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 14:01:33.791130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:33.791094 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 14:01:34.415895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:34.415865 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:01:34.542014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:34.538647 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:34.785206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:34.785127 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:01:34.905489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:34.905460 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:01:36.087141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.087102 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd"] Nov 02 14:01:36.091104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.091086 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.117222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.117194 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd"] Nov 02 14:01:36.128195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.128162 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4c6741-serving-cert\"" Nov 02 14:01:36.192169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.192132 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d28c5\" (UniqueName: \"kubernetes.io/projected/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-kube-api-access-d28c5\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.192367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.192343 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.192463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.192391 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-metrics-certs\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.192532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.192505 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-maas-api-tls\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.263465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.263428 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:36.293579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.293538 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-metrics-certs\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.293786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.293648 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-maas-api-tls\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.293786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.293710 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d28c5\" (UniqueName: \"kubernetes.io/projected/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-kube-api-access-d28c5\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.293938 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.293923 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.294559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.294530 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.296098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.296070 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-metrics-certs\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.296183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.296113 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-maas-api-tls\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.319701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.319671 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d28c5\" (UniqueName: \"kubernetes.io/projected/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-kube-api-access-d28c5\") pod \"maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.409521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.408997 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:36.485008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.467947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:36.829709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.829681 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd"] Nov 02 14:01:36.830601 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:01:36.830579 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd8f0fd_21fe_4e98_a185_f7a4e5036396.slice/crio-b866ff1983aa0664934dea9bde5f3935696511dd6f1a22ea90592c454a81c3bd WatchSource:0}: Error finding container b866ff1983aa0664934dea9bde5f3935696511dd6f1a22ea90592c454a81c3bd: Status 404 returned error can't find the container with id b866ff1983aa0664934dea9bde5f3935696511dd6f1a22ea90592c454a81c3bd Nov 02 14:01:36.919433 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:36.919393 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" event={"ID":"5fd8f0fd-21fe-4e98-a185-f7a4e5036396","Type":"ContainerStarted","Data":"b866ff1983aa0664934dea9bde5f3935696511dd6f1a22ea90592c454a81c3bd"} Nov 02 14:01:37.767441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:37.767409 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:01:37.944182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:37.944118 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" event={"ID":"5fd8f0fd-21fe-4e98-a185-f7a4e5036396","Type":"ContainerStarted","Data":"0ffc0cba9b2c021282f72a603bc82cb3c82cabdff30c41823105e694fe43cbdf"} Nov 02 14:01:37.944388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:37.944276 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:37.991934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:37.991877 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" podStartSLOduration=1.5091204280000001 podStartE2EDuration="1.991860227s" totalImagesPullingTime="482.739799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:37.990722083 +0000 UTC m=+1046.716343863" watchObservedRunningTime="2026-11-02 14:01:37.991860227 +0000 UTC m=+1046.717482039" Nov 02 14:01:38.150008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:38.144119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:39.783857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:39.783826 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:01:39.973507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:39.973469 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:40.149409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:40.148380 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:40.310404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:40.310367 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:41.887008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:41.876738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:43.555008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:43.553160 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:43.955579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:43.955551 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:01:45.271970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:45.266605 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:46.943403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:46.943332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:48.681749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:48.675610 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:50.292378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:50.292333 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:01:50.339725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:50.339688 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:52.316083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:52.316046 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:53.953079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:53.953026 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:55.712892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:55.712842 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:57.397825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:57.397791 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:59.145264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:59.145205 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:59.530436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:01:59.530324 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:00.159036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:00.158976 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:00.633852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:00.633801 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:00.810344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:00.806383 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:00.977094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:00.976898 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:01.122703 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:02:01.122667 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:02:01.262079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:01.261920 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:01.897770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:01.897723 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:02.199070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.198951 5591 generic.go:356] "Generic (PLEG): container finished" podID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerID="ba4db4d3bc4e2081554ae3daeef27910d4de9e203d5ccf4ef79a843f04c382e1" exitCode=0 Nov 02 14:02:02.199070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.199034 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" event={"ID":"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c","Type":"ContainerDied","Data":"ba4db4d3bc4e2081554ae3daeef27910d4de9e203d5ccf4ef79a843f04c382e1"} Nov 02 14:02:02.199511 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.199495 5591 scope.go:122] "RemoveContainer" containerID="ba4db4d3bc4e2081554ae3daeef27910d4de9e203d5ccf4ef79a843f04c382e1" Nov 02 14:02:02.201750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.201674 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:02.201866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.201848 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:02.201967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.201948 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:02.264132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.264099 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4"] Nov 02 14:02:02.268621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.268594 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.294831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.294796 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4"] Nov 02 14:02:02.296116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.296093 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5e9a9d-openshift-default-dockercfg-ph2cj\"" Nov 02 14:02:02.361700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.361659 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:02.397502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397463 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-credential-socket\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.397502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397504 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-podinfo\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.397767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397586 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-envoy\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.397767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397617 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xx27\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-kube-api-access-8xx27\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.397767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397674 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istiod-ca-cert\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.397767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397735 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-certs\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.398043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397806 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-data\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.398043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397845 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-token\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.398043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.397910 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-socket\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.498836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.498748 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-token\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.498836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.498823 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-socket\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.498850 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-credential-socket\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.498867 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-podinfo\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.498893 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-envoy\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.498916 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xx27\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-kube-api-access-8xx27\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.499131 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istiod-ca-cert\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.499293 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-certs\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.499323 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-socket\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.499455 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-data\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.499619 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-credential-socket\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.499651 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-certs\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.499850 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-data\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.499929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.499854 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istiod-ca-cert\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.501439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.501417 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-envoy\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.501679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.501651 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-podinfo\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.554019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.538916 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:02.560237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.555015 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xx27\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-kube-api-access-8xx27\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.572446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.555513 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-token\") pod \"e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.627579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.627306 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:02.758063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:02.754364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:03.207161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:03.207129 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" event={"ID":"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c","Type":"ContainerStarted","Data":"73db9a5e7a365aa3de23874d1fdcd7af815d0eb82f50f7a0cfd8388b977d954a"} Nov 02 14:02:03.265545 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:02:03.265514 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12c6bcd_c1d0_41a1_9c01_5b0a0f78b0c0.slice/crio-a549c12f4e3373bc12a1024c13c37012da5bf6ec097c119ee243cf835984637e WatchSource:0}: Error finding container a549c12f4e3373bc12a1024c13c37012da5bf6ec097c119ee243cf835984637e: Status 404 returned error can't find the container with id a549c12f4e3373bc12a1024c13c37012da5bf6ec097c119ee243cf835984637e Nov 02 14:02:03.268297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:03.268263 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:03.268393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:03.268329 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:03.268393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:03.268359 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:03.277764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:03.277741 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4"] Nov 02 14:02:04.025325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.025290 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:02:04.025509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.025473 5591 patch_prober.go:29] interesting pod/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 14:02:04.025550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.025515 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 14:02:04.214746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.214706 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" event={"ID":"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0","Type":"ContainerStarted","Data":"d9d209b87ebecd51e14eb77b0128bd897f933c9aef7273d571574ddcf2a5b60d"} Nov 02 14:02:04.214926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.214770 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" event={"ID":"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0","Type":"ContainerStarted","Data":"a549c12f4e3373bc12a1024c13c37012da5bf6ec097c119ee243cf835984637e"} Nov 02 14:02:04.290243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.290141 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" podStartSLOduration=2.290128429 podStartE2EDuration="2.290128429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:04.287487478 +0000 UTC m=+1073.013109245" watchObservedRunningTime="2026-11-02 14:02:04.290128429 +0000 UTC m=+1073.015750258" Nov 02 14:02:04.534452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.534304 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:04.632537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.632507 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:04.646615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.646580 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:04.809491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.809456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:04.992332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:04.992247 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:05.024805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:05.024777 5591 patch_prober.go:29] interesting pod/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 14:02:05.024952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:05.024825 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 14:02:05.221485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:05.221446 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:05.222373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:05.222354 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:05.280330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:05.280246 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:06.024921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:06.024888 5591 patch_prober.go:29] interesting pod/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 14:02:06.025339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:06.024944 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 14:02:06.475479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:06.475428 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:06.662569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:06.662527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:06.852650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:06.852548 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:07.029307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:07.029269 5591 patch_prober.go:29] interesting pod/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 14:02:07.029901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:07.029864 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 14:02:07.035517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:07.035435 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:08.024867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.024834 5591 patch_prober.go:29] interesting pod/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 14:02:08.025084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.024893 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 14:02:08.562464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.562428 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:08.683227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.683176 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n"] Nov 02 14:02:08.695000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.694961 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:08.709996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.709951 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n"] Nov 02 14:02:08.720962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.720931 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5e9a9d-dockercfg-nhhkc\"" Nov 02 14:02:08.721884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.721855 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n\" (UID: \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:08.722124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.722100 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krdxm\" (UniqueName: \"kubernetes.io/projected/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-kube-api-access-krdxm\") pod \"payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n\" (UID: \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:08.728089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.728020 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:08.795006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.785478 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5e9a9d\"" Nov 02 14:02:08.879114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.835940 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdxm\" (UniqueName: \"kubernetes.io/projected/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-kube-api-access-krdxm\") pod \"payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n\" (UID: \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:08.902009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.879619 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n\" (UID: \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:08.902454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.884975 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n\" (UID: \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:08.933035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:08.923809 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krdxm\" (UniqueName: \"kubernetes.io/projected/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-kube-api-access-krdxm\") pod \"payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n\" (UID: \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:09.011756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:09.011720 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:09.049664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:09.042487 5591 patch_prober.go:29] interesting pod/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 14:02:09.049664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:09.042544 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 14:02:09.413620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:09.413587 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n"] Nov 02 14:02:09.414823 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:02:09.414792 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6a4a12_5b32_4c58_9e2f_53aef438a1bf.slice/crio-4190f731a6068d24152a0ec9954c42550294034100c46159684d5bbd4e4846af WatchSource:0}: Error finding container 4190f731a6068d24152a0ec9954c42550294034100c46159684d5bbd4e4846af: Status 404 returned error can't find the container with id 4190f731a6068d24152a0ec9954c42550294034100c46159684d5bbd4e4846af Nov 02 14:02:09.709806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:09.709711 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:02:10.030271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:10.030198 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:02:10.277124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:10.277086 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" event={"ID":"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf","Type":"ContainerStarted","Data":"ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba"} Nov 02 14:02:10.277313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:10.277143 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" event={"ID":"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf","Type":"ContainerStarted","Data":"4190f731a6068d24152a0ec9954c42550294034100c46159684d5bbd4e4846af"} Nov 02 14:02:10.277313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:10.277171 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:10.278600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:10.278576 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:02:10.323737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:10.323635 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" podStartSLOduration=2.323621061 podStartE2EDuration="2.323621061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:10.321834104 +0000 UTC m=+1079.047455874" watchObservedRunningTime="2026-11-02 14:02:10.323621061 +0000 UTC m=+1079.049242830" Nov 02 14:02:10.553218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:10.553185 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:11.463589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:11.463554 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:12.266007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:12.258093 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:13.893016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:13.885874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:15.808586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:15.806087 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:17.764502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:17.760480 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:19.342059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:19.342020 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:20.915525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:20.915493 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:21.080173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:21.080136 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:21.286166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:21.286093 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:02:21.485927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:21.485887 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:22.706540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:22.706502 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:24.237524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.237490 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh"] Nov 02 14:02:24.243215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.243189 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.272003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.271023 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5e9a9d-serving-cert\"" Nov 02 14:02:24.276296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.276267 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh"] Nov 02 14:02:24.329716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.329684 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-metrics-certs\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.329889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.329744 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfz2x\" (UniqueName: \"kubernetes.io/projected/0cfea36d-ed38-45ef-a056-2e6091d929ab-kube-api-access-cfz2x\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.329941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.329902 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfea36d-ed38-45ef-a056-2e6091d929ab-openshift-service-ca\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.330065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.330047 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-maas-api-tls\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.374807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.374753 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:24.451007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.436655 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfea36d-ed38-45ef-a056-2e6091d929ab-openshift-service-ca\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.451518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.451481 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-maas-api-tls\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.451839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.451820 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-metrics-certs\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.452026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.452008 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfz2x\" (UniqueName: \"kubernetes.io/projected/0cfea36d-ed38-45ef-a056-2e6091d929ab-kube-api-access-cfz2x\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.453567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.453540 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfea36d-ed38-45ef-a056-2e6091d929ab-openshift-service-ca\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.515050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.507681 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-metrics-certs\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.515050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.512396 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-maas-api-tls\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.515050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.514599 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfz2x\" (UniqueName: \"kubernetes.io/projected/0cfea36d-ed38-45ef-a056-2e6091d929ab-kube-api-access-cfz2x\") pod \"maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.540788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.539485 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:24.563564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.563531 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:24.711429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.711398 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:24.878228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:24.878198 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh"] Nov 02 14:02:24.878920 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:02:24.878866 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfea36d_ed38_45ef_a056_2e6091d929ab.slice/crio-15edd6159b768e3e37d455a71a30bb7ef8b47b42cd3a11845f57031cb4dde729 WatchSource:0}: Error finding container 15edd6159b768e3e37d455a71a30bb7ef8b47b42cd3a11845f57031cb4dde729: Status 404 returned error can't find the container with id 15edd6159b768e3e37d455a71a30bb7ef8b47b42cd3a11845f57031cb4dde729 Nov 02 14:02:25.591532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:25.591406 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" event={"ID":"0cfea36d-ed38-45ef-a056-2e6091d929ab","Type":"ContainerStarted","Data":"d1294c2ea0429b71ac90ded2de7ab6785b0d5a9ec52c941c84c133f625089fd1"} Nov 02 14:02:25.592178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:25.592140 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" event={"ID":"0cfea36d-ed38-45ef-a056-2e6091d929ab","Type":"ContainerStarted","Data":"15edd6159b768e3e37d455a71a30bb7ef8b47b42cd3a11845f57031cb4dde729"} Nov 02 14:02:25.605067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:25.601946 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:25.670048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:25.669959 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" podStartSLOduration=1.29507072 podStartE2EDuration="1.669938775s" totalImagesPullingTime="374.868055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:25.666882426 +0000 UTC m=+1094.392504199" watchObservedRunningTime="2026-11-02 14:02:25.669938775 +0000 UTC m=+1094.395560546" Nov 02 14:02:26.263557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:26.263517 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:27.866006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:27.865335 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:29.484769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:29.484717 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:31.074412 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:02:31.074360 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:02:31.102282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:31.102244 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:31.366562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:31.354850 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:31.635241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:31.635208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:31.956191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:31.956070 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:32.609930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:32.609903 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:02:33.280009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:33.279837 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:35.187916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:35.187865 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:35.833001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:35.832958 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:36.095523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:36.095433 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:37.475510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.475463 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:37.552043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.552012 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:02:37.574899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.574867 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2"] Nov 02 14:02:37.579493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.579473 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:37.598652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.598600 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-584171-dockercfg-m5v8m\"" Nov 02 14:02:37.598652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.598607 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-584171\"" Nov 02 14:02:37.600456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.600428 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2"] Nov 02 14:02:37.621314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.621280 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/06d32887-02fc-4bef-a410-fbcac62f3397-kube-api-access-8xbfr\") pod \"payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2\" (UID: \"06d32887-02fc-4bef-a410-fbcac62f3397\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:37.623517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.623072 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d32887-02fc-4bef-a410-fbcac62f3397-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2\" (UID: \"06d32887-02fc-4bef-a410-fbcac62f3397\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:37.747064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.729208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:37.813895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.813840 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d32887-02fc-4bef-a410-fbcac62f3397-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2\" (UID: \"06d32887-02fc-4bef-a410-fbcac62f3397\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:37.814162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.814058 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/06d32887-02fc-4bef-a410-fbcac62f3397-kube-api-access-8xbfr\") pod \"payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2\" (UID: \"06d32887-02fc-4bef-a410-fbcac62f3397\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:37.819775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.819663 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d32887-02fc-4bef-a410-fbcac62f3397-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2\" (UID: \"06d32887-02fc-4bef-a410-fbcac62f3397\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:37.963315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:37.963038 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/06d32887-02fc-4bef-a410-fbcac62f3397-kube-api-access-8xbfr\") pod \"payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2\" (UID: \"06d32887-02fc-4bef-a410-fbcac62f3397\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:38.195185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:38.195152 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:39.015233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:39.015200 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2"] Nov 02 14:02:39.016647 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:02:39.016622 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d32887_02fc_4bef_a410_fbcac62f3397.slice/crio-e1141f23f324f4339a976f2595b696c3d7609ca624274dd2f418ec09fd945de3 WatchSource:0}: Error finding container e1141f23f324f4339a976f2595b696c3d7609ca624274dd2f418ec09fd945de3: Status 404 returned error can't find the container with id e1141f23f324f4339a976f2595b696c3d7609ca624274dd2f418ec09fd945de3 Nov 02 14:02:39.638008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:39.636580 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:39.815772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:39.815723 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" event={"ID":"06d32887-02fc-4bef-a410-fbcac62f3397","Type":"ContainerStarted","Data":"b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5"} Nov 02 14:02:39.816011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:39.815791 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:39.816011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:39.815811 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" event={"ID":"06d32887-02fc-4bef-a410-fbcac62f3397","Type":"ContainerStarted","Data":"e1141f23f324f4339a976f2595b696c3d7609ca624274dd2f418ec09fd945de3"} Nov 02 14:02:39.860563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:39.860510 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" podStartSLOduration=2.860492198 podStartE2EDuration="2.860492198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:39.857754143 +0000 UTC m=+1108.583375914" watchObservedRunningTime="2026-11-02 14:02:39.860492198 +0000 UTC m=+1108.586113967" Nov 02 14:02:41.350598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:41.350559 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:43.055010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:43.052566 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:43.764860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:43.764817 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8"] Nov 02 14:02:43.957680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:43.957638 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:44.134004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:44.131521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:44.966118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:44.966076 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:45.373635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:45.373544 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:46.832740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:46.832678 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:47.217514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:47.217467 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:47.417674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:47.417639 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:48.371015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.370970 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:48.653042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.647531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:48.765479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.765432 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" podUID="21a0ac73-db13-4d3c-a0a7-239c4b2d722c" containerName="payload-processing" containerID="cri-o://dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85" gracePeriod=25 Nov 02 14:02:48.917000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.916907 5591 generic.go:356] "Generic (PLEG): container finished" podID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerID="d9d209b87ebecd51e14eb77b0128bd897f933c9aef7273d571574ddcf2a5b60d" exitCode=0 Nov 02 14:02:48.917158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.917000 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" event={"ID":"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0","Type":"ContainerDied","Data":"d9d209b87ebecd51e14eb77b0128bd897f933c9aef7273d571574ddcf2a5b60d"} Nov 02 14:02:48.917547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.917530 5591 scope.go:122] "RemoveContainer" containerID="d9d209b87ebecd51e14eb77b0128bd897f933c9aef7273d571574ddcf2a5b60d" Nov 02 14:02:48.919232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.919204 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:48.919308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.919274 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:48.919345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:48.919309 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:02:49.111413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:49.111381 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:49.270286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:49.270213 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:49.538785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:49.538678 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" podUID="21a0ac73-db13-4d3c-a0a7-239c4b2d722c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:02:49.926202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:49.926168 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" event={"ID":"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0","Type":"ContainerStarted","Data":"c0400d1763049b2fdcfd5d20a5e0a80c09c6bc14c283bd2179bf89e794a510f0"} Nov 02 14:02:50.221816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:50.221735 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:50.347010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:50.343803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:50.628773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:50.628685 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:50.629969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:50.629944 5591 patch_prober.go:29] interesting pod/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:02:50.630124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:50.630032 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:02:50.825829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:50.825797 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:02:51.629006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:51.628947 5591 patch_prober.go:29] interesting pod/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:02:51.629438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:51.629034 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:02:52.015010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:52.008040 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:52.223344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:52.223308 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc"] Nov 02 14:02:52.260724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:52.260684 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk"] Nov 02 14:02:52.428164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:52.428129 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:52.622126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:52.622092 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:52.629477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:52.629444 5591 patch_prober.go:29] interesting pod/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:02:52.629848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:52.629504 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:02:53.628948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.628913 5591 patch_prober.go:29] interesting pod/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:02:53.629147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.629007 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:02:53.661169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.661131 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b"] Nov 02 14:02:53.664823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.664795 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b"] Nov 02 14:02:53.664976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.664902 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.673312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.673285 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:53.680444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.680420 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-584171-serving-cert\"" Nov 02 14:02:53.762025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.761948 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-maas-api-tls\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.762208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.762137 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-metrics-certs\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.762277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.762256 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abecc48f-a96b-4b62-bc15-7db3200599de-openshift-service-ca\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.762391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.762373 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhkj\" (UniqueName: \"kubernetes.io/projected/abecc48f-a96b-4b62-bc15-7db3200599de-kube-api-access-5xhkj\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.857492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.857445 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:53.864000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.863959 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-metrics-certs\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.864148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.864068 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abecc48f-a96b-4b62-bc15-7db3200599de-openshift-service-ca\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.864202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.864169 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhkj\" (UniqueName: \"kubernetes.io/projected/abecc48f-a96b-4b62-bc15-7db3200599de-kube-api-access-5xhkj\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.864274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.864260 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-maas-api-tls\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.864827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.864801 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abecc48f-a96b-4b62-bc15-7db3200599de-openshift-service-ca\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.866550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.866517 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-metrics-certs\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.866792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.866771 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-maas-api-tls\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.915677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.913779 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhkj\" (UniqueName: \"kubernetes.io/projected/abecc48f-a96b-4b62-bc15-7db3200599de-kube-api-access-5xhkj\") pod \"maas-api-e2e-api-a-584171-7dc768f859-8864b\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:53.991006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:53.990408 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:54.064017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:54.054661 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:54.393034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:54.393000 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b"] Nov 02 14:02:54.393307 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:02:54.393284 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabecc48f_a96b_4b62_bc15_7db3200599de.slice/crio-4a5d0217eeba192dade34028e81e8a154dd7fc87e2241bddc1c938b66d7cbfd1 WatchSource:0}: Error finding container 4a5d0217eeba192dade34028e81e8a154dd7fc87e2241bddc1c938b66d7cbfd1: Status 404 returned error can't find the container with id 4a5d0217eeba192dade34028e81e8a154dd7fc87e2241bddc1c938b66d7cbfd1 Nov 02 14:02:54.630012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:54.629171 5591 patch_prober.go:29] interesting pod/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:02:54.630012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:54.629251 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:02:55.075961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:55.075920 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" event={"ID":"abecc48f-a96b-4b62-bc15-7db3200599de","Type":"ContainerStarted","Data":"4a5d0217eeba192dade34028e81e8a154dd7fc87e2241bddc1c938b66d7cbfd1"} Nov 02 14:02:55.174501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:55.174449 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:55.350625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:55.350534 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:55.513494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:55.513461 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:55.632912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:55.632881 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:55.633862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:55.633839 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:02:55.762008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:55.755627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:56.090247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:56.090152 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" event={"ID":"abecc48f-a96b-4b62-bc15-7db3200599de","Type":"ContainerStarted","Data":"345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931"} Nov 02 14:02:56.090816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:56.090792 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:02:56.135111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:56.135047 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" podStartSLOduration=2.555049176 podStartE2EDuration="3.135027347s" totalImagesPullingTime="579.978171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:56.133893942 +0000 UTC m=+1124.859515711" watchObservedRunningTime="2026-11-02 14:02:56.135027347 +0000 UTC m=+1124.860649118" Nov 02 14:02:57.224603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.224561 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" podUID="0d5caf66-7ef3-4f97-8c22-f2c0fef0e170" containerName="payload-processing" containerID="cri-o://c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722" gracePeriod=25 Nov 02 14:02:57.261135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.261065 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" podUID="6cb583eb-b763-42f5-bc28-e651ae7e8d23" containerName="payload-pre-processing" containerID="cri-o://ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946" gracePeriod=25 Nov 02 14:02:57.557277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.557247 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:02:57.663347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.663262 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:57.671231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.671203 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:02:57.716791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.716754 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-kube-api-access-qvvtt\" (UniqueName: \"kubernetes.io/projected/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-kube-api-access-qvvtt\") pod \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\" (UID: \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\") " Nov 02 14:02:57.717120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.717098 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-plugins-config-volume\") pod \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\" (UID: \"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170\") " Nov 02 14:02:57.720976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.720947 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-plugins-config-volume" pod "0d5caf66-7ef3-4f97-8c22-f2c0fef0e170" (UID: "0d5caf66-7ef3-4f97-8c22-f2c0fef0e170"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:02:57.724219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.724189 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-kube-api-access-qvvtt" pod "0d5caf66-7ef3-4f97-8c22-f2c0fef0e170" (UID: "0d5caf66-7ef3-4f97-8c22-f2c0fef0e170"). InnerVolumeSpecName "kube-api-access-qvvtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:02:57.819295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.819202 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb583eb-b763-42f5-bc28-e651ae7e8d23-kube-api-access-4zxfr\" (UniqueName: \"kubernetes.io/projected/6cb583eb-b763-42f5-bc28-e651ae7e8d23-kube-api-access-4zxfr\") pod \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\" (UID: \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\") " Nov 02 14:02:57.819451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.819413 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb583eb-b763-42f5-bc28-e651ae7e8d23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb583eb-b763-42f5-bc28-e651ae7e8d23-plugins-config-volume\") pod \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\" (UID: \"6cb583eb-b763-42f5-bc28-e651ae7e8d23\") " Nov 02 14:02:57.819778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.819739 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb583eb-b763-42f5-bc28-e651ae7e8d23-plugins-config-volume" pod "6cb583eb-b763-42f5-bc28-e651ae7e8d23" (UID: "6cb583eb-b763-42f5-bc28-e651ae7e8d23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:02:57.819995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.819962 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvvtt\" (UniqueName: \"kubernetes.io/projected/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-kube-api-access-qvvtt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:02:57.820063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.820001 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb583eb-b763-42f5-bc28-e651ae7e8d23-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:02:57.820063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.820015 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:02:57.821212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.821192 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb583eb-b763-42f5-bc28-e651ae7e8d23-kube-api-access-4zxfr" pod "6cb583eb-b763-42f5-bc28-e651ae7e8d23" (UID: "6cb583eb-b763-42f5-bc28-e651ae7e8d23"). InnerVolumeSpecName "kube-api-access-4zxfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:02:57.921190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:57.921156 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zxfr\" (UniqueName: \"kubernetes.io/projected/6cb583eb-b763-42f5-bc28-e651ae7e8d23-kube-api-access-4zxfr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:02:58.104370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.104289 5591 generic.go:356] "Generic (PLEG): container finished" podID="6cb583eb-b763-42f5-bc28-e651ae7e8d23" containerID="ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946" exitCode=0 Nov 02 14:02:58.104370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.104332 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" Nov 02 14:02:58.104571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.104371 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" event={"ID":"6cb583eb-b763-42f5-bc28-e651ae7e8d23","Type":"ContainerDied","Data":"ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946"} Nov 02 14:02:58.104571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.104420 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk" event={"ID":"6cb583eb-b763-42f5-bc28-e651ae7e8d23","Type":"ContainerDied","Data":"32f35bc2685c045178b190e38d4cc7d688a89b6417903ab1167c35eb9698bd60"} Nov 02 14:02:58.104571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.104447 5591 scope.go:122] "RemoveContainer" containerID="ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946" Nov 02 14:02:58.105849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.105827 5591 generic.go:356] "Generic (PLEG): container finished" podID="0d5caf66-7ef3-4f97-8c22-f2c0fef0e170" containerID="c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722" exitCode=0 Nov 02 14:02:58.105962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.105870 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" Nov 02 14:02:58.105962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.105904 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" event={"ID":"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170","Type":"ContainerDied","Data":"c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722"} Nov 02 14:02:58.105962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.105953 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc" event={"ID":"0d5caf66-7ef3-4f97-8c22-f2c0fef0e170","Type":"ContainerDied","Data":"ba3e16583108524c6870dbf2d191d7a0eb157257e7c8ce8ff09c4d229d4550aa"} Nov 02 14:02:58.115917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.115901 5591 scope.go:122] "RemoveContainer" containerID="ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946" Nov 02 14:02:58.116208 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:02:58.116191 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946\": container with ID starting with ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946 not found: ID does not exist" containerID="ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946" Nov 02 14:02:58.116295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.116219 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946"} err="failed to get container status \"ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946\": rpc error: code = NotFound desc = could not find container \"ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946\": container with ID starting with ef4d0b78eb9a7e93d01c247bece60007e3bdca3c3555c8fa6af2c8fefcbde946 not found: ID does not exist" Nov 02 14:02:58.116295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.116242 5591 scope.go:122] "RemoveContainer" containerID="c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722" Nov 02 14:02:58.125892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.125875 5591 scope.go:122] "RemoveContainer" containerID="c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722" Nov 02 14:02:58.126143 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:02:58.126122 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722\": container with ID starting with c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722 not found: ID does not exist" containerID="c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722" Nov 02 14:02:58.126203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.126151 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722"} err="failed to get container status \"c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722\": rpc error: code = NotFound desc = could not find container \"c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722\": container with ID starting with c33d6f75e898b510e59cc0fc5611da674a36e1aa05a587775e97ce31dbb61722 not found: ID does not exist" Nov 02 14:02:58.159692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.159663 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc"] Nov 02 14:02:58.172197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.172172 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c6741-67f9795bd5-87gmc"] Nov 02 14:02:58.199903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.199870 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk"] Nov 02 14:02:58.206170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:58.206140 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c6741-7db76df9cf-hcgqk"] Nov 02 14:02:59.465447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:59.465408 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:59.482622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:59.482577 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5caf66-7ef3-4f97-8c22-f2c0fef0e170" path="/var/lib/kubelet/pods/0d5caf66-7ef3-4f97-8c22-f2c0fef0e170/volumes" Nov 02 14:02:59.483774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:59.483739 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb583eb-b763-42f5-bc28-e651ae7e8d23" path="/var/lib/kubelet/pods/6cb583eb-b763-42f5-bc28-e651ae7e8d23/volumes" Nov 02 14:02:59.541007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:59.539089 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" podUID="21a0ac73-db13-4d3c-a0a7-239c4b2d722c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:02:59.627163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:02:59.627127 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:01.040010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:01.039953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:01.074132 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:03:01.074099 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:03:01.223527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:01.219912 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:01.782744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:01.782707 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:01.946893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:01.946857 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:02.102722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:02.102648 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:03:02.844092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:02.844055 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:04.471004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:04.461018 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:03:04.541638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:04.541604 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:06.122473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:06.122395 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:06.447008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:06.445536 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd"] Nov 02 14:03:06.447008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:06.445759 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" podUID="5fd8f0fd-21fe-4e98-a185-f7a4e5036396" containerName="maas-api" containerID="cri-o://0ffc0cba9b2c021282f72a603bc82cb3c82cabdff30c41823105e694fe43cbdf" gracePeriod=30 Nov 02 14:03:06.538717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:06.538558 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:06.764097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:06.764036 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:07.044009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:07.002467 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:07.368096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:07.367997 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:03:07.416814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:07.416778 5591 generic.go:356] "Generic (PLEG): container finished" podID="5fd8f0fd-21fe-4e98-a185-f7a4e5036396" containerID="0ffc0cba9b2c021282f72a603bc82cb3c82cabdff30c41823105e694fe43cbdf" exitCode=0 Nov 02 14:03:07.417279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:07.417255 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" event={"ID":"5fd8f0fd-21fe-4e98-a185-f7a4e5036396","Type":"ContainerDied","Data":"0ffc0cba9b2c021282f72a603bc82cb3c82cabdff30c41823105e694fe43cbdf"} Nov 02 14:03:07.469760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:07.469710 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:03:07.518932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:07.518888 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:03:07.566206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:07.566142 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:03:07.921456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:07.917485 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:03:08.018019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.011524 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:08.220889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.220855 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:03:08.314733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.314692 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-kube-api-access-d28c5\" (UniqueName: \"kubernetes.io/projected/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-kube-api-access-d28c5\") pod \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " Nov 02 14:03:08.314929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.314763 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-openshift-service-ca\") pod \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " Nov 02 14:03:08.314929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.314801 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-metrics-certs\") pod \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " Nov 02 14:03:08.314929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.314833 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-maas-api-tls\") pod \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\" (UID: \"5fd8f0fd-21fe-4e98-a185-f7a4e5036396\") " Nov 02 14:03:08.315278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.315241 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-openshift-service-ca" pod "5fd8f0fd-21fe-4e98-a185-f7a4e5036396" (UID: "5fd8f0fd-21fe-4e98-a185-f7a4e5036396"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:03:08.316060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.315724 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:08.318320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.317563 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-kube-api-access-d28c5" pod "5fd8f0fd-21fe-4e98-a185-f7a4e5036396" (UID: "5fd8f0fd-21fe-4e98-a185-f7a4e5036396"). InnerVolumeSpecName "kube-api-access-d28c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:03:08.318320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.318198 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-maas-api-tls" pod "5fd8f0fd-21fe-4e98-a185-f7a4e5036396" (UID: "5fd8f0fd-21fe-4e98-a185-f7a4e5036396"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:03:08.318613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.318583 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-metrics-certs" pod "5fd8f0fd-21fe-4e98-a185-f7a4e5036396" (UID: "5fd8f0fd-21fe-4e98-a185-f7a4e5036396"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:03:08.457068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.429640 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d28c5\" (UniqueName: \"kubernetes.io/projected/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-kube-api-access-d28c5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:08.457068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.429681 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:08.457068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.429695 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd8f0fd-21fe-4e98-a185-f7a4e5036396-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:08.506116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.506057 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" event={"ID":"5fd8f0fd-21fe-4e98-a185-f7a4e5036396","Type":"ContainerDied","Data":"b866ff1983aa0664934dea9bde5f3935696511dd6f1a22ea90592c454a81c3bd"} Nov 02 14:03:08.506323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.506154 5591 scope.go:122] "RemoveContainer" containerID="0ffc0cba9b2c021282f72a603bc82cb3c82cabdff30c41823105e694fe43cbdf" Nov 02 14:03:08.506393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.506347 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd" Nov 02 14:03:08.615740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.607958 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd"] Nov 02 14:03:08.616714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:08.616679 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c6741-77dfc968fb-b8ftd"] Nov 02 14:03:09.076645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.076616 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:03:09.170656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.170618 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-plugins-config-volume\") pod \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\" (UID: \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\") " Nov 02 14:03:09.170813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.170727 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-kube-api-access-28x5j\" (UniqueName: \"kubernetes.io/projected/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-kube-api-access-28x5j\") pod \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\" (UID: \"21a0ac73-db13-4d3c-a0a7-239c4b2d722c\") " Nov 02 14:03:09.171094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.171064 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-plugins-config-volume" pod "21a0ac73-db13-4d3c-a0a7-239c4b2d722c" (UID: "21a0ac73-db13-4d3c-a0a7-239c4b2d722c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:03:09.171359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.171343 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:09.172829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.172808 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-kube-api-access-28x5j" pod "21a0ac73-db13-4d3c-a0a7-239c4b2d722c" (UID: "21a0ac73-db13-4d3c-a0a7-239c4b2d722c"). InnerVolumeSpecName "kube-api-access-28x5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:03:09.273224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.273190 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28x5j\" (UniqueName: \"kubernetes.io/projected/21a0ac73-db13-4d3c-a0a7-239c4b2d722c-kube-api-access-28x5j\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:09.433352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.433310 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd8f0fd-21fe-4e98-a185-f7a4e5036396" path="/var/lib/kubelet/pods/5fd8f0fd-21fe-4e98-a185-f7a4e5036396/volumes" Nov 02 14:03:09.515697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.515663 5591 generic.go:356] "Generic (PLEG): container finished" podID="21a0ac73-db13-4d3c-a0a7-239c4b2d722c" containerID="dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85" exitCode=0 Nov 02 14:03:09.516164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.515710 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" event={"ID":"21a0ac73-db13-4d3c-a0a7-239c4b2d722c","Type":"ContainerDied","Data":"dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85"} Nov 02 14:03:09.516164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.515756 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" event={"ID":"21a0ac73-db13-4d3c-a0a7-239c4b2d722c","Type":"ContainerDied","Data":"91051cee8b8cac501d911717c5187f3250ca71005af3b081a1aa2eed54fa0a72"} Nov 02 14:03:09.516164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.515785 5591 scope.go:122] "RemoveContainer" containerID="dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85" Nov 02 14:03:09.516164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.515712 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8" Nov 02 14:03:09.530008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.529969 5591 scope.go:122] "RemoveContainer" containerID="dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85" Nov 02 14:03:09.530505 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:03:09.530450 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85\": container with ID starting with dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85 not found: ID does not exist" containerID="dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85" Nov 02 14:03:09.530505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.530488 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85"} err="failed to get container status \"dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85\": rpc error: code = NotFound desc = could not find container \"dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85\": container with ID starting with dc4a018da021e0fe58b3b34a685b76dd6fc5c9375da826ed169aac23e4a0eb85 not found: ID does not exist" Nov 02 14:03:09.569630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.569597 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8"] Nov 02 14:03:09.582574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.582541 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-dp7k8"] Nov 02 14:03:09.816661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:09.816580 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:10.632017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:10.631093 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:11.217412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:11.217382 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:11.473529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:11.473444 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a0ac73-db13-4d3c-a0a7-239c4b2d722c" path="/var/lib/kubelet/pods/21a0ac73-db13-4d3c-a0a7-239c4b2d722c/volumes" Nov 02 14:03:12.397009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:12.393603 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:14.360008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:14.359709 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:14.555433 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:14.555390 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:15.098520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:15.098484 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:15.289169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:15.289131 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:15.831119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:15.831076 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:16.162072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.162034 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:16.431130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.427195 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm"] Nov 02 14:03:16.431357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431233 5591 cpu_manager.go:414] "containerMap: removing container" podUID="21a0ac73-db13-4d3c-a0a7-239c4b2d722c" containerName="payload-processing" Nov 02 14:03:16.431357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431257 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a0ac73-db13-4d3c-a0a7-239c4b2d722c" containerName="payload-processing" Nov 02 14:03:16.431357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431285 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6cb583eb-b763-42f5-bc28-e651ae7e8d23" containerName="payload-pre-processing" Nov 02 14:03:16.431357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431293 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb583eb-b763-42f5-bc28-e651ae7e8d23" containerName="payload-pre-processing" Nov 02 14:03:16.431357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431328 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0d5caf66-7ef3-4f97-8c22-f2c0fef0e170" containerName="payload-processing" Nov 02 14:03:16.431357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431342 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5caf66-7ef3-4f97-8c22-f2c0fef0e170" containerName="payload-processing" Nov 02 14:03:16.431357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431353 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5fd8f0fd-21fe-4e98-a185-f7a4e5036396" containerName="maas-api" Nov 02 14:03:16.431705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431361 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd8f0fd-21fe-4e98-a185-f7a4e5036396" containerName="maas-api" Nov 02 14:03:16.431705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431580 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5caf66-7ef3-4f97-8c22-f2c0fef0e170" containerName="payload-processing" Nov 02 14:03:16.431705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431591 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd8f0fd-21fe-4e98-a185-f7a4e5036396" containerName="maas-api" Nov 02 14:03:16.431705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431609 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a0ac73-db13-4d3c-a0a7-239c4b2d722c" containerName="payload-processing" Nov 02 14:03:16.431705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.431622 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb583eb-b763-42f5-bc28-e651ae7e8d23" containerName="payload-pre-processing" Nov 02 14:03:16.510831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.510755 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:16.578891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.559427 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:16.703242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.693767 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm"] Nov 02 14:03:16.806569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.793836 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d776122a-dockercfg-7tptq\"" Nov 02 14:03:16.806569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.793922 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d776122a\"" Nov 02 14:03:16.839007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.822043 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e21615-83ba-46fa-b632-b225333e56a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm\" (UID: \"34e21615-83ba-46fa-b632-b225333e56a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:16.840790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.840250 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8kj\" (UniqueName: \"kubernetes.io/projected/34e21615-83ba-46fa-b632-b225333e56a1-kube-api-access-fr8kj\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm\" (UID: \"34e21615-83ba-46fa-b632-b225333e56a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:16.864780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.864736 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:16.946103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.946061 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e21615-83ba-46fa-b632-b225333e56a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm\" (UID: \"34e21615-83ba-46fa-b632-b225333e56a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:16.946465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.946443 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8kj\" (UniqueName: \"kubernetes.io/projected/34e21615-83ba-46fa-b632-b225333e56a1-kube-api-access-fr8kj\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm\" (UID: \"34e21615-83ba-46fa-b632-b225333e56a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:16.947064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.947038 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e21615-83ba-46fa-b632-b225333e56a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm\" (UID: \"34e21615-83ba-46fa-b632-b225333e56a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:16.978879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:16.978806 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8kj\" (UniqueName: \"kubernetes.io/projected/34e21615-83ba-46fa-b632-b225333e56a1-kube-api-access-fr8kj\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm\" (UID: \"34e21615-83ba-46fa-b632-b225333e56a1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:17.140895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.140859 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:17.260125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.260048 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7"] Nov 02 14:03:17.266453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.266425 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.286810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.286786 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8d5eec-openshift-default-dockercfg-sn2xx\"" Nov 02 14:03:17.305562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.305534 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7"] Nov 02 14:03:17.352850 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.352821 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-podinfo\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.353088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.352868 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-credential-socket\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.353088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.352979 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b339ca8-336c-4696-a390-64c9a1ba582a-istiod-ca-cert\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.353088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.353036 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-data\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.353088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.353065 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-certs\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.353304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.353184 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8d6\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-kube-api-access-tt8d6\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.353304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.353225 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-envoy\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.353304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.353294 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-socket\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.353452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.353332 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-token\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.462007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.461856 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8d6\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-kube-api-access-tt8d6\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.462007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.461923 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-envoy\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.497015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.494288 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-socket\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.497015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.494879 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-socket\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.508718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.507133 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-token\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.508718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.507300 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-podinfo\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.508718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.507355 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-credential-socket\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.508718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.507418 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b339ca8-336c-4696-a390-64c9a1ba582a-istiod-ca-cert\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.508718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.507451 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-data\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.508718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.507504 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-certs\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.508718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.507996 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-certs\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.523056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.513594 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b339ca8-336c-4696-a390-64c9a1ba582a-istiod-ca-cert\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.523056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.513820 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-credential-socket\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.523056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.514036 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-data\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.543008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.524754 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-envoy\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.543473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.543427 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8d6\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-kube-api-access-tt8d6\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.547414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.547355 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-podinfo\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.569149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.567739 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-token\") pod \"e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.607967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.603317 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:17.632322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.630913 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm"] Nov 02 14:03:17.689114 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:03:17.678413 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e21615_83ba_46fa_b632_b225333e56a1.slice/crio-e87a23246be2d1170a065815b2fe5e9f60ac686a67aba5bdbd18497ad09d4a45 WatchSource:0}: Error finding container e87a23246be2d1170a065815b2fe5e9f60ac686a67aba5bdbd18497ad09d4a45: Status 404 returned error can't find the container with id e87a23246be2d1170a065815b2fe5e9f60ac686a67aba5bdbd18497ad09d4a45 Nov 02 14:03:17.883829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.883798 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" event={"ID":"34e21615-83ba-46fa-b632-b225333e56a1","Type":"ContainerStarted","Data":"ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd"} Nov 02 14:03:17.884292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.883852 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" event={"ID":"34e21615-83ba-46fa-b632-b225333e56a1","Type":"ContainerStarted","Data":"e87a23246be2d1170a065815b2fe5e9f60ac686a67aba5bdbd18497ad09d4a45"} Nov 02 14:03:17.884292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.883878 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:17.940051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:17.939997 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" podStartSLOduration=1.939966572 podStartE2EDuration="1.939966572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:17.938260258 +0000 UTC m=+1146.663882027" watchObservedRunningTime="2026-11-02 14:03:17.939966572 +0000 UTC m=+1146.665588342" Nov 02 14:03:18.173310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:18.151099 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7"] Nov 02 14:03:18.177387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:18.177351 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:03:18.177578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:18.177563 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:03:18.177664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:18.177653 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:03:18.448192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:18.448094 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:18.661008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:18.657602 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:18.893121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:18.893087 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" event={"ID":"0b339ca8-336c-4696-a390-64c9a1ba582a","Type":"ContainerStarted","Data":"1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363"} Nov 02 14:03:18.893588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:18.893569 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" event={"ID":"0b339ca8-336c-4696-a390-64c9a1ba582a","Type":"ContainerStarted","Data":"c119dd7115fef31a75f956fbc052d11e626adea032165a3f4a9dede789252552"} Nov 02 14:03:19.605194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:19.605161 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:19.611563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:19.611536 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:19.714810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:19.714757 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" podStartSLOduration=2.714741359 podStartE2EDuration="2.714741359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:18.948415232 +0000 UTC m=+1147.674037002" watchObservedRunningTime="2026-11-02 14:03:19.714741359 +0000 UTC m=+1148.440363181" Nov 02 14:03:19.907853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:19.907817 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:19.908606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:19.908586 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:03:20.569549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:20.561702 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:20.762690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:20.762654 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:21.150009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:21.143198 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:21.552406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:21.552320 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:21.590890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:21.590858 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s"] Nov 02 14:03:21.591170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:21.591143 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" containerID="cri-o://73db9a5e7a365aa3de23874d1fdcd7af815d0eb82f50f7a0cfd8388b977d954a" gracePeriod=30 Nov 02 14:03:21.769965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:21.769912 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:22.429880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:22.429841 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:22.810277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:22.801256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:24.382244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:24.382212 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:24.562011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:24.561964 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:24.758056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:24.751019 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:25.266767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:25.266724 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:25.280263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:25.280228 5591 patch_prober.go:29] interesting pod/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:03:25.280410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:25.280293 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:03:25.446168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:25.446138 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:26.513540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:26.513504 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:26.683766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:26.683713 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:26.985204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:26.985151 5591 generic.go:356] "Generic (PLEG): container finished" podID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerID="73db9a5e7a365aa3de23874d1fdcd7af815d0eb82f50f7a0cfd8388b977d954a" exitCode=0 Nov 02 14:03:26.985402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:26.985221 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" event={"ID":"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c","Type":"ContainerDied","Data":"73db9a5e7a365aa3de23874d1fdcd7af815d0eb82f50f7a0cfd8388b977d954a"} Nov 02 14:03:26.985402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:26.985318 5591 scope.go:122] "RemoveContainer" containerID="ba4db4d3bc4e2081554ae3daeef27910d4de9e203d5ccf4ef79a843f04c382e1" Nov 02 14:03:27.422225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.422201 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:03:27.493422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.492671 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-certs\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.493422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.492753 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-token\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-token\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.493422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.492902 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-socket\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.521545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.521110 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-certs" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:27.543042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.532120 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-envoy\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.543042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.532302 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istiod-ca-cert\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.543042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.532368 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-kube-api-access-bfbmb\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-kube-api-access-bfbmb\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.543042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.532445 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-credential-socket\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.543042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.532483 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-data\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.543042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.532535 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-podinfo\") pod \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\" (UID: \"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c\") " Nov 02 14:03:27.543042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.533315 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.545213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.545178 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-socket" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:27.605518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.558928 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istiod-ca-cert" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:03:27.605518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.559086 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-credential-socket" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:27.605518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.559251 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-data" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:27.605518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.598964 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-token" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:03:27.612222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.612180 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-podinfo" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:03:27.626322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.624809 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-envoy" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:27.626322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.626280 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-kube-api-access-bfbmb" pod "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" (UID: "f7de0a8a-dd54-41d3-9444-d8a6e6667f2c"). InnerVolumeSpecName "kube-api-access-bfbmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:03:27.650373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.649617 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.650373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.649649 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.650373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.649664 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfbmb\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-kube-api-access-bfbmb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.650373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.649679 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.650373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.649692 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.650373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.649706 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.650373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.649720 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.650373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.649732 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:27.993915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.993881 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" Nov 02 14:03:27.994126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.993922 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s" event={"ID":"f7de0a8a-dd54-41d3-9444-d8a6e6667f2c","Type":"ContainerDied","Data":"8263908f2bd1d58cc9f253a0813379e8fbe3cf716282b72148c6a7c5b3508f4c"} Nov 02 14:03:27.994126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:27.993998 5591 scope.go:122] "RemoveContainer" containerID="73db9a5e7a365aa3de23874d1fdcd7af815d0eb82f50f7a0cfd8388b977d954a" Nov 02 14:03:28.049480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:28.049447 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s"] Nov 02 14:03:28.062865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:28.062840 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9eb67b-openshift-default-8557847dcf-p6z7s"] Nov 02 14:03:28.380731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:28.380644 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:28.897122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:28.897092 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:03:29.362904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:29.362812 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n"] Nov 02 14:03:29.439823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:29.439791 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" path="/var/lib/kubelet/pods/f7de0a8a-dd54-41d3-9444-d8a6e6667f2c/volumes" Nov 02 14:03:29.528041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:29.528000 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:29.703614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:29.703410 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:30.299023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:30.294666 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:30.541609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:30.541566 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:31.074141 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:03:31.074107 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:03:31.752244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:31.752205 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:32.093127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:32.088802 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:34.022030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.021953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:34.023059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.022860 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:03:34.364029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.363912 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" podUID="2a6a4a12-5b32-4c58-9e2f-53aef438a1bf" containerName="payload-pre-processing" containerID="cri-o://ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba" gracePeriod=25 Nov 02 14:03:34.803998 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.803963 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:03:34.938772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.938731 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-plugins-config-volume\") pod \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\" (UID: \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\") " Nov 02 14:03:34.938966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.938859 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-kube-api-access-krdxm\" (UniqueName: \"kubernetes.io/projected/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-kube-api-access-krdxm\") pod \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\" (UID: \"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf\") " Nov 02 14:03:34.939140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.939117 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-plugins-config-volume" pod "2a6a4a12-5b32-4c58-9e2f-53aef438a1bf" (UID: "2a6a4a12-5b32-4c58-9e2f-53aef438a1bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:03:34.939518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.939498 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:34.940896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:34.940876 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-kube-api-access-krdxm" pod "2a6a4a12-5b32-4c58-9e2f-53aef438a1bf" (UID: "2a6a4a12-5b32-4c58-9e2f-53aef438a1bf"). InnerVolumeSpecName "kube-api-access-krdxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:03:35.040936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.040835 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krdxm\" (UniqueName: \"kubernetes.io/projected/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf-kube-api-access-krdxm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:35.111999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.111949 5591 generic.go:356] "Generic (PLEG): container finished" podID="2a6a4a12-5b32-4c58-9e2f-53aef438a1bf" containerID="ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba" exitCode=0 Nov 02 14:03:35.112147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.112023 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" Nov 02 14:03:35.112147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.112035 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" event={"ID":"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf","Type":"ContainerDied","Data":"ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba"} Nov 02 14:03:35.112147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.112094 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n" event={"ID":"2a6a4a12-5b32-4c58-9e2f-53aef438a1bf","Type":"ContainerDied","Data":"4190f731a6068d24152a0ec9954c42550294034100c46159684d5bbd4e4846af"} Nov 02 14:03:35.112147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.112123 5591 scope.go:122] "RemoveContainer" containerID="ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba" Nov 02 14:03:35.131660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.131638 5591 scope.go:122] "RemoveContainer" containerID="ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba" Nov 02 14:03:35.131934 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:03:35.131913 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba\": container with ID starting with ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba not found: ID does not exist" containerID="ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba" Nov 02 14:03:35.132023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.131944 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba"} err="failed to get container status \"ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba\": rpc error: code = NotFound desc = could not find container \"ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba\": container with ID starting with ca11392487b085159d6e89b7607c7e1b8738d5fb684a631362360c50619377ba not found: ID does not exist" Nov 02 14:03:35.182240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.182206 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n"] Nov 02 14:03:35.194556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.194529 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5e9a9d-7d5f8869f9-5xw9n"] Nov 02 14:03:35.431688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.431654 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6a4a12-5b32-4c58-9e2f-53aef438a1bf" path="/var/lib/kubelet/pods/2a6a4a12-5b32-4c58-9e2f-53aef438a1bf/volumes" Nov 02 14:03:35.803064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:35.787854 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:37.528970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:37.528936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:37.712579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:37.712495 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:39.231239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:39.231203 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:39.416321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:39.416235 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:40.782765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:40.782730 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:41.515042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:41.513524 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:42.976257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:42.976218 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh"] Nov 02 14:03:42.976770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:42.976491 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" podUID="0cfea36d-ed38-45ef-a056-2e6091d929ab" containerName="maas-api" containerID="cri-o://d1294c2ea0429b71ac90ded2de7ab6785b0d5a9ec52c941c84c133f625089fd1" gracePeriod=30 Nov 02 14:03:43.097601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.097565 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:43.397057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.397004 5591 generic.go:356] "Generic (PLEG): container finished" podID="0cfea36d-ed38-45ef-a056-2e6091d929ab" containerID="d1294c2ea0429b71ac90ded2de7ab6785b0d5a9ec52c941c84c133f625089fd1" exitCode=0 Nov 02 14:03:43.397386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.397360 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" event={"ID":"0cfea36d-ed38-45ef-a056-2e6091d929ab","Type":"ContainerDied","Data":"d1294c2ea0429b71ac90ded2de7ab6785b0d5a9ec52c941c84c133f625089fd1"} Nov 02 14:03:43.544027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.541090 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:03:43.635006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.632216 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-metrics-certs\") pod \"0cfea36d-ed38-45ef-a056-2e6091d929ab\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " Nov 02 14:03:43.635006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.632328 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cfea36d-ed38-45ef-a056-2e6091d929ab-kube-api-access-cfz2x\" (UniqueName: \"kubernetes.io/projected/0cfea36d-ed38-45ef-a056-2e6091d929ab-kube-api-access-cfz2x\") pod \"0cfea36d-ed38-45ef-a056-2e6091d929ab\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " Nov 02 14:03:43.635006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.632428 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cfea36d-ed38-45ef-a056-2e6091d929ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfea36d-ed38-45ef-a056-2e6091d929ab-openshift-service-ca\") pod \"0cfea36d-ed38-45ef-a056-2e6091d929ab\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " Nov 02 14:03:43.635006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.632480 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-maas-api-tls\") pod \"0cfea36d-ed38-45ef-a056-2e6091d929ab\" (UID: \"0cfea36d-ed38-45ef-a056-2e6091d929ab\") " Nov 02 14:03:43.636129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.636096 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cfea36d-ed38-45ef-a056-2e6091d929ab-openshift-service-ca" pod "0cfea36d-ed38-45ef-a056-2e6091d929ab" (UID: "0cfea36d-ed38-45ef-a056-2e6091d929ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:03:43.650689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.637917 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-maas-api-tls" pod "0cfea36d-ed38-45ef-a056-2e6091d929ab" (UID: "0cfea36d-ed38-45ef-a056-2e6091d929ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:03:43.658697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.653855 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-metrics-certs" pod "0cfea36d-ed38-45ef-a056-2e6091d929ab" (UID: "0cfea36d-ed38-45ef-a056-2e6091d929ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:03:43.683013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.678151 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfea36d-ed38-45ef-a056-2e6091d929ab-kube-api-access-cfz2x" pod "0cfea36d-ed38-45ef-a056-2e6091d929ab" (UID: "0cfea36d-ed38-45ef-a056-2e6091d929ab"). InnerVolumeSpecName "kube-api-access-cfz2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:03:43.734611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.734575 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfz2x\" (UniqueName: \"kubernetes.io/projected/0cfea36d-ed38-45ef-a056-2e6091d929ab-kube-api-access-cfz2x\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:43.734611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.734608 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cfea36d-ed38-45ef-a056-2e6091d929ab-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:43.734828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.734619 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:43.734828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.734628 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cfea36d-ed38-45ef-a056-2e6091d929ab-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:43.799004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:43.798425 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:44.405403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:44.405372 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" Nov 02 14:03:44.405844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:44.405407 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh" event={"ID":"0cfea36d-ed38-45ef-a056-2e6091d929ab","Type":"ContainerDied","Data":"15edd6159b768e3e37d455a71a30bb7ef8b47b42cd3a11845f57031cb4dde729"} Nov 02 14:03:44.405844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:44.405464 5591 scope.go:122] "RemoveContainer" containerID="d1294c2ea0429b71ac90ded2de7ab6785b0d5a9ec52c941c84c133f625089fd1" Nov 02 14:03:44.494369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:44.494332 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh"] Nov 02 14:03:44.507044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:44.507013 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5e9a9d-c74d9b4ff-x8xfh"] Nov 02 14:03:45.068165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:45.067216 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:45.248433 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:45.248402 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:45.430684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:45.430654 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cfea36d-ed38-45ef-a056-2e6091d929ab" path="/var/lib/kubelet/pods/0cfea36d-ed38-45ef-a056-2e6091d929ab/volumes" Nov 02 14:03:45.912432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:45.912399 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:46.100885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:46.100856 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:46.303448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:46.303360 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4"] Nov 02 14:03:46.303661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:46.303630 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" containerID="cri-o://c0400d1763049b2fdcfd5d20a5e0a80c09c6bc14c283bd2179bf89e794a510f0" gracePeriod=30 Nov 02 14:03:46.476007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:46.471449 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:47.943676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:47.943639 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:48.182855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:48.182814 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:49.202014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:49.191246 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:49.376757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:49.376714 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:50.211673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:50.211636 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:50.396451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:50.396412 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:50.607087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:50.602611 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:51.480777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.480744 5591 generic.go:356] "Generic (PLEG): container finished" podID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerID="c0400d1763049b2fdcfd5d20a5e0a80c09c6bc14c283bd2179bf89e794a510f0" exitCode=0 Nov 02 14:03:51.481276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.480838 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" event={"ID":"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0","Type":"ContainerDied","Data":"c0400d1763049b2fdcfd5d20a5e0a80c09c6bc14c283bd2179bf89e794a510f0"} Nov 02 14:03:51.481276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.480923 5591 scope.go:122] "RemoveContainer" containerID="d9d209b87ebecd51e14eb77b0128bd897f933c9aef7273d571574ddcf2a5b60d" Nov 02 14:03:51.582438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.582379 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:03:51.601312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601285 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-socket\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601343 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-token\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-token\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601386 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istiod-ca-cert\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601423 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-podinfo\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601504 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-kube-api-access-8xx27\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-kube-api-access-8xx27\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601537 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-certs\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601539 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-socket" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:51.601733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601646 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-credential-socket\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601677 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-data\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601705 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-envoy\") pod \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\" (UID: \"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0\") " Nov 02 14:03:51.601900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601848 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-certs" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:51.601900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.601876 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istiod-ca-cert" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:03:51.602125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.602099 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-data" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:51.602456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.602431 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:51.602550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.602462 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:51.602550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.602478 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:51.602550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.602491 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:51.602550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.602502 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-credential-socket" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:51.603677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.603651 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-token" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:03:51.603837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.603809 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-kube-api-access-8xx27" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "kube-api-access-8xx27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:03:51.603837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.603818 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-podinfo" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:03:51.604055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.604037 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-envoy" pod "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" (UID: "c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:03:51.703622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.703583 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:51.703622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.703615 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:51.703622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.703626 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:51.703849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.703634 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:51.703849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:51.703644 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xx27\" (UniqueName: \"kubernetes.io/projected/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0-kube-api-access-8xx27\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:03:52.197448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:52.197406 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:52.488033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:52.487928 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" Nov 02 14:03:52.488033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:52.487943 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4" event={"ID":"c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0","Type":"ContainerDied","Data":"a549c12f4e3373bc12a1024c13c37012da5bf6ec097c119ee243cf835984637e"} Nov 02 14:03:52.488033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:52.488010 5591 scope.go:122] "RemoveContainer" containerID="c0400d1763049b2fdcfd5d20a5e0a80c09c6bc14c283bd2179bf89e794a510f0" Nov 02 14:03:52.556325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:52.556292 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4"] Nov 02 14:03:52.568813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:52.568790 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5e9a9d-openshift-default-7777955bd9-75pv4"] Nov 02 14:03:53.432688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:53.432652 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" path="/var/lib/kubelet/pods/c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0/volumes" Nov 02 14:03:53.979783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:53.979748 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:55.641523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:55.641484 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:57.377030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:57.365602 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:58.878772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:58.878702 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:59.205937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:59.205892 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:59.373421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:59.373382 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:03:59.542505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:03:59.542414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:01.087968 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:01.087932 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:04:01.463009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:01.456838 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:01.915881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:01.915815 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:02.462311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.462282 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7036-7388f7"] Nov 02 14:04:02.463026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463001 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" Nov 02 14:04:02.463026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463024 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" Nov 02 14:04:02.463219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463044 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0cfea36d-ed38-45ef-a056-2e6091d929ab" containerName="maas-api" Nov 02 14:04:02.463219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463050 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cfea36d-ed38-45ef-a056-2e6091d929ab" containerName="maas-api" Nov 02 14:04:02.463219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463063 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2a6a4a12-5b32-4c58-9e2f-53aef438a1bf" containerName="payload-pre-processing" Nov 02 14:04:02.463219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463069 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6a4a12-5b32-4c58-9e2f-53aef438a1bf" containerName="payload-pre-processing" Nov 02 14:04:02.463219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463102 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" Nov 02 14:04:02.463219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463107 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" Nov 02 14:04:02.463219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463116 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" Nov 02 14:04:02.463219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463122 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" Nov 02 14:04:02.463541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463225 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" Nov 02 14:04:02.463541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463237 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7de0a8a-dd54-41d3-9444-d8a6e6667f2c" containerName="istio-proxy" Nov 02 14:04:02.463541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463252 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" Nov 02 14:04:02.463541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463259 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" Nov 02 14:04:02.463541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463266 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cfea36d-ed38-45ef-a056-2e6091d929ab" containerName="maas-api" Nov 02 14:04:02.463541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.463273 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6a4a12-5b32-4c58-9e2f-53aef438a1bf" containerName="payload-pre-processing" Nov 02 14:04:02.466655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.466638 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-7388f7" Nov 02 14:04:02.483712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.483681 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7036-7388f7"] Nov 02 14:04:02.493190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.493162 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7jzh4\"" Nov 02 14:04:02.614708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.614670 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzb5\" (UniqueName: \"kubernetes.io/projected/cd02e591-c811-4fa2-a19a-94329e9cec8b-kube-api-access-czzb5\") pod \"test-curl-7036-7388f7\" (UID: \"cd02e591-c811-4fa2-a19a-94329e9cec8b\") " pod="opendatahub/test-curl-7036-7388f7" Nov 02 14:04:02.716213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.716112 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzb5\" (UniqueName: \"kubernetes.io/projected/cd02e591-c811-4fa2-a19a-94329e9cec8b-kube-api-access-czzb5\") pod \"test-curl-7036-7388f7\" (UID: \"cd02e591-c811-4fa2-a19a-94329e9cec8b\") " pod="opendatahub/test-curl-7036-7388f7" Nov 02 14:04:02.752972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.752921 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzb5\" (UniqueName: \"kubernetes.io/projected/cd02e591-c811-4fa2-a19a-94329e9cec8b-kube-api-access-czzb5\") pod \"test-curl-7036-7388f7\" (UID: \"cd02e591-c811-4fa2-a19a-94329e9cec8b\") " pod="opendatahub/test-curl-7036-7388f7" Nov 02 14:04:02.780489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:02.780455 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-7388f7" Nov 02 14:04:03.108490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:03.108463 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7036-7388f7"] Nov 02 14:04:03.109666 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:04:03.109640 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd02e591_c811_4fa2_a19a_94329e9cec8b.slice/crio-e8e9afcc043456c6c0949dbf6007477e51dacaccec2dd8dbaecf944417e6f76b WatchSource:0}: Error finding container e8e9afcc043456c6c0949dbf6007477e51dacaccec2dd8dbaecf944417e6f76b: Status 404 returned error can't find the container with id e8e9afcc043456c6c0949dbf6007477e51dacaccec2dd8dbaecf944417e6f76b Nov 02 14:04:03.173024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:03.170552 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:03.370685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:03.370577 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:03.598118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:03.598087 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7036-7388f7" event={"ID":"cd02e591-c811-4fa2-a19a-94329e9cec8b","Type":"ContainerStarted","Data":"e8e9afcc043456c6c0949dbf6007477e51dacaccec2dd8dbaecf944417e6f76b"} Nov 02 14:04:04.870069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:04.863922 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:06.545131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:06.545058 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:06.642541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:06.642501 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7036-7388f7" event={"ID":"cd02e591-c811-4fa2-a19a-94329e9cec8b","Type":"ContainerStarted","Data":"ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270"} Nov 02 14:04:06.683701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:06.683645 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7036-7388f7" podStartSLOduration=1.766297545 podStartE2EDuration="4.683632157s" totalImagesPullingTime="2.917334612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:06.683074212 +0000 UTC m=+1195.408695982" watchObservedRunningTime="2026-11-02 14:04:06.683632157 +0000 UTC m=+1195.409253926" Nov 02 14:04:07.602127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:07.602091 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7036-7388f7"] Nov 02 14:04:07.605698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:07.605667 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7036-7388f7"] Nov 02 14:04:08.176544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:08.176478 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:08.386573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:08.386537 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:08.658748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:08.658706 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7036-7388f7" podUID="cd02e591-c811-4fa2-a19a-94329e9cec8b" containerName="test-curl-7036-7388f7" containerID="cri-o://ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270" gracePeriod=2 Nov 02 14:04:08.673502 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:08.673460 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7036-7388f7\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="cd02e591-c811-4fa2-a19a-94329e9cec8b" pod="opendatahub/test-curl-7036-7388f7" Nov 02 14:04:09.667962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.667923 5591 generic.go:356] "Generic (PLEG): container finished" podID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerID="1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363" exitCode=0 Nov 02 14:04:09.668572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.668291 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" event={"ID":"0b339ca8-336c-4696-a390-64c9a1ba582a","Type":"ContainerDied","Data":"1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363"} Nov 02 14:04:09.669007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.668974 5591 scope.go:122] "RemoveContainer" containerID="1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363" Nov 02 14:04:09.670524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.670494 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:09.670633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.670560 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:09.670633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.670593 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:09.787244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.787169 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2"] Nov 02 14:04:09.807545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.807514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:09.977543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:09.977496 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:10.150343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:10.150314 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:10.681318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:10.681294 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" event={"ID":"0b339ca8-336c-4696-a390-64c9a1ba582a","Type":"ContainerStarted","Data":"c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819"} Nov 02 14:04:10.909814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:10.909790 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-7388f7" Nov 02 14:04:10.986531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:10.986443 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd02e591-c811-4fa2-a19a-94329e9cec8b-kube-api-access-czzb5\" (UniqueName: \"kubernetes.io/projected/cd02e591-c811-4fa2-a19a-94329e9cec8b-kube-api-access-czzb5\") pod \"cd02e591-c811-4fa2-a19a-94329e9cec8b\" (UID: \"cd02e591-c811-4fa2-a19a-94329e9cec8b\") " Nov 02 14:04:10.988555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:10.988528 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd02e591-c811-4fa2-a19a-94329e9cec8b-kube-api-access-czzb5" pod "cd02e591-c811-4fa2-a19a-94329e9cec8b" (UID: "cd02e591-c811-4fa2-a19a-94329e9cec8b"). InnerVolumeSpecName "kube-api-access-czzb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:04:11.087772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.087733 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czzb5\" (UniqueName: \"kubernetes.io/projected/cd02e591-c811-4fa2-a19a-94329e9cec8b-kube-api-access-czzb5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:11.479957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.479921 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd02e591-c811-4fa2-a19a-94329e9cec8b" path="/var/lib/kubelet/pods/cd02e591-c811-4fa2-a19a-94329e9cec8b/volumes" Nov 02 14:04:11.604425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.604394 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:04:11.604649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.604620 5591 patch_prober.go:29] interesting pod/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 14:04:11.604734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.604689 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 14:04:11.682009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.677411 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:11.728724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.728645 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:04:11.771668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.771586 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:04:11.773342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.773036 5591 generic.go:356] "Generic (PLEG): container finished" podID="cd02e591-c811-4fa2-a19a-94329e9cec8b" containerID="ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270" exitCode=137 Nov 02 14:04:11.781951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.773085 5591 scope.go:122] "RemoveContainer" containerID="ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270" Nov 02 14:04:11.785860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.773171 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-7388f7" Nov 02 14:04:11.828565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.828013 5591 scope.go:122] "RemoveContainer" containerID="ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270" Nov 02 14:04:11.828840 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:11.828665 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270\": container with ID starting with ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270 not found: ID does not exist" containerID="ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270" Nov 02 14:04:11.828840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:11.828701 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270"} err="failed to get container status \"ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270\": rpc error: code = NotFound desc = could not find container \"ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270\": container with ID starting with ea0b4e25520687d5ff1a8a2c1138203fd07be9839435f32bdf6f79469d04d270 not found: ID does not exist" Nov 02 14:04:12.605391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:12.605359 5591 patch_prober.go:29] interesting pod/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 14:04:12.605573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:12.605426 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 14:04:13.203367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:13.203323 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk"] Nov 02 14:04:13.382917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:13.382874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:13.562028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:13.561923 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:13.604579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:13.604544 5591 patch_prober.go:29] interesting pod/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 14:04:13.604751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:13.604601 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 14:04:14.560174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.560138 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:14.604568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.604536 5591 patch_prober.go:29] interesting pod/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 14:04:14.604757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.604586 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 14:04:14.787900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.787843 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" podUID="06d32887-02fc-4bef-a410-fbcac62f3397" containerName="payload-pre-processing" containerID="cri-o://b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5" gracePeriod=25 Nov 02 14:04:14.990254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.990220 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz"] Nov 02 14:04:14.990919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.990901 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" Nov 02 14:04:14.990919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.990919 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12c6bcd-c1d0-41a1-9c01-5b0a0f78b0c0" containerName="istio-proxy" Nov 02 14:04:14.991080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.990950 5591 cpu_manager.go:414] "containerMap: removing container" podUID="cd02e591-c811-4fa2-a19a-94329e9cec8b" containerName="test-curl-7036-7388f7" Nov 02 14:04:14.991080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.990956 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd02e591-c811-4fa2-a19a-94329e9cec8b" containerName="test-curl-7036-7388f7" Nov 02 14:04:14.991080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.991063 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd02e591-c811-4fa2-a19a-94329e9cec8b" containerName="test-curl-7036-7388f7" Nov 02 14:04:14.995367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:14.995344 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.067059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.054620 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e543e0-openshift-default-dockercfg-k5vqp\"" Nov 02 14:04:15.067059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.060939 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz"] Nov 02 14:04:15.090728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.090678 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-data\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.091516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.091240 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-certs\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.092131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.092110 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-credential-socket\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.092354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.092327 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgxg\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-kube-api-access-djgxg\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.092617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.092588 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-envoy\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.092909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.092881 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-socket\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.093642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.093592 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-podinfo\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.093886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.093866 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33351e87-01da-44ee-98c8-e59f6e0aa709-istiod-ca-cert\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.094223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.094203 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-token\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197099 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-credential-socket\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197147 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgxg\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-kube-api-access-djgxg\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197210 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-envoy\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197296 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-socket\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197348 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-podinfo\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197416 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33351e87-01da-44ee-98c8-e59f6e0aa709-istiod-ca-cert\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197518 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-token\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197584 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-data\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.209458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.197613 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-certs\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.241974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.217624 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:15.241974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.218535 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33351e87-01da-44ee-98c8-e59f6e0aa709-istiod-ca-cert\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.241974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.220640 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-data\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.241974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.233595 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-credential-socket\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.241974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.233541 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-socket\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.259113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.253788 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-podinfo\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.307789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.307736 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-certs\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.364056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.357691 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-envoy\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.382015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.378566 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:04:15.390020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.388857 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgxg\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-kube-api-access-djgxg\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.443712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.436523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:15.459554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.459521 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-token\") pod \"e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.553115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.538761 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d32887-02fc-4bef-a410-fbcac62f3397-kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/06d32887-02fc-4bef-a410-fbcac62f3397-kube-api-access-8xbfr\") pod \"06d32887-02fc-4bef-a410-fbcac62f3397\" (UID: \"06d32887-02fc-4bef-a410-fbcac62f3397\") " Nov 02 14:04:15.553115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.538811 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d32887-02fc-4bef-a410-fbcac62f3397-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d32887-02fc-4bef-a410-fbcac62f3397-plugins-config-volume\") pod \"06d32887-02fc-4bef-a410-fbcac62f3397\" (UID: \"06d32887-02fc-4bef-a410-fbcac62f3397\") " Nov 02 14:04:15.571070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.567370 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d32887-02fc-4bef-a410-fbcac62f3397-plugins-config-volume" pod "06d32887-02fc-4bef-a410-fbcac62f3397" (UID: "06d32887-02fc-4bef-a410-fbcac62f3397"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:04:15.624664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.624577 5591 patch_prober.go:29] interesting pod/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 14:04:15.624898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.624872 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 14:04:15.628779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.628748 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:15.666024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.641429 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d32887-02fc-4bef-a410-fbcac62f3397-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:15.698536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.698507 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d32887-02fc-4bef-a410-fbcac62f3397-kube-api-access-8xbfr" pod "06d32887-02fc-4bef-a410-fbcac62f3397" (UID: "06d32887-02fc-4bef-a410-fbcac62f3397"). InnerVolumeSpecName "kube-api-access-8xbfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:04:15.792546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.792511 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/06d32887-02fc-4bef-a410-fbcac62f3397-kube-api-access-8xbfr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:15.815873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.815842 5591 generic.go:356] "Generic (PLEG): container finished" podID="06d32887-02fc-4bef-a410-fbcac62f3397" containerID="b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5" exitCode=0 Nov 02 14:04:15.816060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.815893 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" event={"ID":"06d32887-02fc-4bef-a410-fbcac62f3397","Type":"ContainerDied","Data":"b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5"} Nov 02 14:04:15.816060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.815897 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" Nov 02 14:04:15.816060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.815951 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2" event={"ID":"06d32887-02fc-4bef-a410-fbcac62f3397","Type":"ContainerDied","Data":"e1141f23f324f4339a976f2595b696c3d7609ca624274dd2f418ec09fd945de3"} Nov 02 14:04:15.816060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.815976 5591 scope.go:122] "RemoveContainer" containerID="b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5" Nov 02 14:04:15.827300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.827280 5591 scope.go:122] "RemoveContainer" containerID="b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5" Nov 02 14:04:15.827636 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:15.827616 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5\": container with ID starting with b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5 not found: ID does not exist" containerID="b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5" Nov 02 14:04:15.827709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.827643 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5"} err="failed to get container status \"b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5\": rpc error: code = NotFound desc = could not find container \"b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5\": container with ID starting with b5443379ccc7b184249cfbd2240c0b0e5dac4dfbec5afd68e4e2a8f494e807f5 not found: ID does not exist" Nov 02 14:04:15.888895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.888862 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2"] Nov 02 14:04:15.892909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:15.892878 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-584171-7b6478fcdc-lbfz2"] Nov 02 14:04:16.036063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.036033 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz"] Nov 02 14:04:16.037047 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:04:16.037013 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33351e87_01da_44ee_98c8_e59f6e0aa709.slice/crio-bcc2eafbed78e358c4b817e6d7e1744a4fbd6050c5ffdff22373ac7f518e11a0 WatchSource:0}: Error finding container bcc2eafbed78e358c4b817e6d7e1744a4fbd6050c5ffdff22373ac7f518e11a0: Status 404 returned error can't find the container with id bcc2eafbed78e358c4b817e6d7e1744a4fbd6050c5ffdff22373ac7f518e11a0 Nov 02 14:04:16.040272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.040234 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:16.040370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.040325 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:16.040410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.040374 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:16.600230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.600197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:16.609572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.609544 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:04:16.610003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.609966 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:04:16.610732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.610704 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:04:16.844046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.843974 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" event={"ID":"33351e87-01da-44ee-98c8-e59f6e0aa709","Type":"ContainerStarted","Data":"cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0"} Nov 02 14:04:16.844246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.844105 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" event={"ID":"33351e87-01da-44ee-98c8-e59f6e0aa709","Type":"ContainerStarted","Data":"bcc2eafbed78e358c4b817e6d7e1744a4fbd6050c5ffdff22373ac7f518e11a0"} Nov 02 14:04:16.891736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.891684 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podStartSLOduration=2.8916681029999998 podStartE2EDuration="2.891668103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:16.8904311 +0000 UTC m=+1205.616052871" watchObservedRunningTime="2026-11-02 14:04:16.891668103 +0000 UTC m=+1205.617289874" Nov 02 14:04:16.952538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:16.952501 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:17.433251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:17.433212 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d32887-02fc-4bef-a410-fbcac62f3397" path="/var/lib/kubelet/pods/06d32887-02fc-4bef-a410-fbcac62f3397/volumes" Nov 02 14:04:17.630342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:17.630302 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:17.635828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:17.635800 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:17.853090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:17.853016 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:17.853961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:17.853937 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:04:18.742977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:18.742937 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:19.388662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:19.388632 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:19.551492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:19.551455 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:20.328697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:20.328649 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:20.561111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:20.561066 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:20.578005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:20.573171 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:04:21.482485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.482444 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:21.571209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.571159 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh"] Nov 02 14:04:21.572353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.572321 5591 cpu_manager.go:414] "containerMap: removing container" podUID="06d32887-02fc-4bef-a410-fbcac62f3397" containerName="payload-pre-processing" Nov 02 14:04:21.572532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.572516 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d32887-02fc-4bef-a410-fbcac62f3397" containerName="payload-pre-processing" Nov 02 14:04:21.572850 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.572832 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d32887-02fc-4bef-a410-fbcac62f3397" containerName="payload-pre-processing" Nov 02 14:04:21.605799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.605768 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:21.635005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.626727 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh"] Nov 02 14:04:21.679362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.659768 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e543e0\"" Nov 02 14:04:21.679362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.660067 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e543e0-dockercfg-s8v67\"" Nov 02 14:04:21.679362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.664665 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb"] Nov 02 14:04:21.679362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.675648 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:21.679362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.678445 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb"] Nov 02 14:04:21.708935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.706881 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428bbec0-8481-4fa8-a023-40b8ad5325e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh\" (UID: \"428bbec0-8481-4fa8-a023-40b8ad5325e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:21.708935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.707105 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxcr\" (UniqueName: \"kubernetes.io/projected/428bbec0-8481-4fa8-a023-40b8ad5325e7-kube-api-access-mnxcr\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh\" (UID: \"428bbec0-8481-4fa8-a023-40b8ad5325e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:21.708935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.708765 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:21.852363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.852272 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngnt\" (UniqueName: \"kubernetes.io/projected/6557f55f-06ff-44ee-93fc-0294f1f00317-kube-api-access-jngnt\") pod \"payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb\" (UID: \"6557f55f-06ff-44ee-93fc-0294f1f00317\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:21.852610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.852588 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428bbec0-8481-4fa8-a023-40b8ad5325e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh\" (UID: \"428bbec0-8481-4fa8-a023-40b8ad5325e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:21.852905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.852887 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxcr\" (UniqueName: \"kubernetes.io/projected/428bbec0-8481-4fa8-a023-40b8ad5325e7-kube-api-access-mnxcr\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh\" (UID: \"428bbec0-8481-4fa8-a023-40b8ad5325e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:21.853136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.853116 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6557f55f-06ff-44ee-93fc-0294f1f00317-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb\" (UID: \"6557f55f-06ff-44ee-93fc-0294f1f00317\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:21.871469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.855455 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428bbec0-8481-4fa8-a023-40b8ad5325e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh\" (UID: \"428bbec0-8481-4fa8-a023-40b8ad5325e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:21.949255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.949210 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:21.953461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.953410 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxcr\" (UniqueName: \"kubernetes.io/projected/428bbec0-8481-4fa8-a023-40b8ad5325e7-kube-api-access-mnxcr\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh\" (UID: \"428bbec0-8481-4fa8-a023-40b8ad5325e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:21.980481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.978185 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngnt\" (UniqueName: \"kubernetes.io/projected/6557f55f-06ff-44ee-93fc-0294f1f00317-kube-api-access-jngnt\") pod \"payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb\" (UID: \"6557f55f-06ff-44ee-93fc-0294f1f00317\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:21.980481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.978486 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6557f55f-06ff-44ee-93fc-0294f1f00317-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb\" (UID: \"6557f55f-06ff-44ee-93fc-0294f1f00317\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:21.980481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:21.979578 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6557f55f-06ff-44ee-93fc-0294f1f00317-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb\" (UID: \"6557f55f-06ff-44ee-93fc-0294f1f00317\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:22.004709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.004678 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngnt\" (UniqueName: \"kubernetes.io/projected/6557f55f-06ff-44ee-93fc-0294f1f00317-kube-api-access-jngnt\") pod \"payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb\" (UID: \"6557f55f-06ff-44ee-93fc-0294f1f00317\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:22.010574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.010548 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:22.102273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.102240 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:22.323650 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:04:22.323616 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428bbec0_8481_4fa8_a023_40b8ad5325e7.slice/crio-0b41af5556d22b77eb9a54c6526083b9742e07a8dc295f5c0371cb6b20f6f0ac WatchSource:0}: Error finding container 0b41af5556d22b77eb9a54c6526083b9742e07a8dc295f5c0371cb6b20f6f0ac: Status 404 returned error can't find the container with id 0b41af5556d22b77eb9a54c6526083b9742e07a8dc295f5c0371cb6b20f6f0ac Nov 02 14:04:22.324683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.324662 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh"] Nov 02 14:04:22.368075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.368041 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b"] Nov 02 14:04:22.368304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.368268 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" podUID="abecc48f-a96b-4b62-bc15-7db3200599de" containerName="maas-api" containerID="cri-o://345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931" gracePeriod=30 Nov 02 14:04:22.433232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.433194 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb"] Nov 02 14:04:22.441651 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:04:22.441624 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6557f55f_06ff_44ee_93fc_0294f1f00317.slice/crio-0fe3b3d3b86d37c722f0b1b9c4e692274ef4934465301357da1dfd4c4f3b1c75 WatchSource:0}: Error finding container 0fe3b3d3b86d37c722f0b1b9c4e692274ef4934465301357da1dfd4c4f3b1c75: Status 404 returned error can't find the container with id 0fe3b3d3b86d37c722f0b1b9c4e692274ef4934465301357da1dfd4c4f3b1c75 Nov 02 14:04:22.488491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.488460 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:22.804010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.792325 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:04:22.824045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.820283 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abecc48f-a96b-4b62-bc15-7db3200599de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abecc48f-a96b-4b62-bc15-7db3200599de-openshift-service-ca\") pod \"abecc48f-a96b-4b62-bc15-7db3200599de\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " Nov 02 14:04:22.824045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.820375 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-metrics-certs\") pod \"abecc48f-a96b-4b62-bc15-7db3200599de\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " Nov 02 14:04:22.824045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.820425 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-maas-api-tls\") pod \"abecc48f-a96b-4b62-bc15-7db3200599de\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " Nov 02 14:04:22.824045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.820543 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abecc48f-a96b-4b62-bc15-7db3200599de-kube-api-access-5xhkj\" (UniqueName: \"kubernetes.io/projected/abecc48f-a96b-4b62-bc15-7db3200599de-kube-api-access-5xhkj\") pod \"abecc48f-a96b-4b62-bc15-7db3200599de\" (UID: \"abecc48f-a96b-4b62-bc15-7db3200599de\") " Nov 02 14:04:22.828005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.827633 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abecc48f-a96b-4b62-bc15-7db3200599de-openshift-service-ca" pod "abecc48f-a96b-4b62-bc15-7db3200599de" (UID: "abecc48f-a96b-4b62-bc15-7db3200599de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:04:22.832012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.829647 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-metrics-certs" pod "abecc48f-a96b-4b62-bc15-7db3200599de" (UID: "abecc48f-a96b-4b62-bc15-7db3200599de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:04:22.832012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.829973 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abecc48f-a96b-4b62-bc15-7db3200599de-kube-api-access-5xhkj" pod "abecc48f-a96b-4b62-bc15-7db3200599de" (UID: "abecc48f-a96b-4b62-bc15-7db3200599de"). InnerVolumeSpecName "kube-api-access-5xhkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:04:22.832012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.830289 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-maas-api-tls" pod "abecc48f-a96b-4b62-bc15-7db3200599de" (UID: "abecc48f-a96b-4b62-bc15-7db3200599de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:04:22.929014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.926205 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhkj\" (UniqueName: \"kubernetes.io/projected/abecc48f-a96b-4b62-bc15-7db3200599de-kube-api-access-5xhkj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:22.929014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.926266 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abecc48f-a96b-4b62-bc15-7db3200599de-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:22.929014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.926278 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:22.929014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:22.926291 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abecc48f-a96b-4b62-bc15-7db3200599de-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:23.101373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.101261 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" event={"ID":"428bbec0-8481-4fa8-a023-40b8ad5325e7","Type":"ContainerStarted","Data":"cd73cc6b36b3a1af0574bfe1af1e1d37e6ae9fdbec1a64cc967a7f9ed2d73476"} Nov 02 14:04:23.101373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.101326 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" event={"ID":"428bbec0-8481-4fa8-a023-40b8ad5325e7","Type":"ContainerStarted","Data":"0b41af5556d22b77eb9a54c6526083b9742e07a8dc295f5c0371cb6b20f6f0ac"} Nov 02 14:04:23.101602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.101387 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:23.103684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.103653 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" event={"ID":"6557f55f-06ff-44ee-93fc-0294f1f00317","Type":"ContainerStarted","Data":"9f5ed4302bbc378853850e627567183acd8342c4620920d57d3a34b449655611"} Nov 02 14:04:23.103794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.103708 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" event={"ID":"6557f55f-06ff-44ee-93fc-0294f1f00317","Type":"ContainerStarted","Data":"0fe3b3d3b86d37c722f0b1b9c4e692274ef4934465301357da1dfd4c4f3b1c75"} Nov 02 14:04:23.104310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.104120 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:23.105414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.105386 5591 generic.go:356] "Generic (PLEG): container finished" podID="abecc48f-a96b-4b62-bc15-7db3200599de" containerID="345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931" exitCode=0 Nov 02 14:04:23.105510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.105426 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" Nov 02 14:04:23.105510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.105432 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" event={"ID":"abecc48f-a96b-4b62-bc15-7db3200599de","Type":"ContainerDied","Data":"345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931"} Nov 02 14:04:23.105510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.105476 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b" event={"ID":"abecc48f-a96b-4b62-bc15-7db3200599de","Type":"ContainerDied","Data":"4a5d0217eeba192dade34028e81e8a154dd7fc87e2241bddc1c938b66d7cbfd1"} Nov 02 14:04:23.105510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.105505 5591 scope.go:122] "RemoveContainer" containerID="345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931" Nov 02 14:04:23.118364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.118340 5591 scope.go:122] "RemoveContainer" containerID="345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931" Nov 02 14:04:23.118704 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:23.118680 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931\": container with ID starting with 345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931 not found: ID does not exist" containerID="345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931" Nov 02 14:04:23.118788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.118719 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931"} err="failed to get container status \"345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931\": rpc error: code = NotFound desc = could not find container \"345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931\": container with ID starting with 345d12b47c2bcf6596027f7aafe42af8c008fd9ef37b0aa000ca124bb2485931 not found: ID does not exist" Nov 02 14:04:23.143790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.143751 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" podStartSLOduration=2.143738514 podStartE2EDuration="2.143738514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:23.142501797 +0000 UTC m=+1211.868123569" watchObservedRunningTime="2026-11-02 14:04:23.143738514 +0000 UTC m=+1211.869360285" Nov 02 14:04:23.182221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.182191 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:23.186964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.186911 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" podStartSLOduration=2.186898207 podStartE2EDuration="2.186898207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:23.18445729 +0000 UTC m=+1211.910079061" watchObservedRunningTime="2026-11-02 14:04:23.186898207 +0000 UTC m=+1211.912519976" Nov 02 14:04:23.223922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.223892 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b"] Nov 02 14:04:23.227650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.227625 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-584171-7dc768f859-8864b"] Nov 02 14:04:23.432457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.432424 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abecc48f-a96b-4b62-bc15-7db3200599de" path="/var/lib/kubelet/pods/abecc48f-a96b-4b62-bc15-7db3200599de/volumes" Nov 02 14:04:23.496762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.496732 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:23.669399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.669356 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:23.870892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:23.870808 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:24.244556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.244522 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh"] Nov 02 14:04:24.245615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.245586 5591 cpu_manager.go:414] "containerMap: removing container" podUID="abecc48f-a96b-4b62-bc15-7db3200599de" containerName="maas-api" Nov 02 14:04:24.245615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.245614 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abecc48f-a96b-4b62-bc15-7db3200599de" containerName="maas-api" Nov 02 14:04:24.245828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.245764 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="abecc48f-a96b-4b62-bc15-7db3200599de" containerName="maas-api" Nov 02 14:04:24.251185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.251161 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.266376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.266349 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-envoy\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.266512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.266389 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-certs\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.266581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.266523 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-socket\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.266630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.266588 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2486b6-a99e-4bee-939e-88263acdd4b7-istiod-ca-cert\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.266725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.266712 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-data\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.266855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.266838 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-token\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.266915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.266874 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-credential-socket\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.267025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.267001 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxpj\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-kube-api-access-frxpj\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.267109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.267058 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-podinfo\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.282398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.282367 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ae366a-openshift-default-dockercfg-kx9rr\"" Nov 02 14:04:24.286284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.286258 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh"] Nov 02 14:04:24.314245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.309786 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:24.368751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.368717 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-data\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.368939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.368784 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-token\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.368939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.368810 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-credential-socket\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.368939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.368848 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxpj\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-kube-api-access-frxpj\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369047 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-podinfo\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369096 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-envoy\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369099 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-data\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369124 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-certs\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369177 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-credential-socket\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369191 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-socket\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369266 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2486b6-a99e-4bee-939e-88263acdd4b7-istiod-ca-cert\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369446 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-certs\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369471 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-socket\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.369894 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.369873 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2486b6-a99e-4bee-939e-88263acdd4b7-istiod-ca-cert\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.371293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.371268 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-envoy\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.371597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.371576 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-podinfo\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.438978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.434599 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-token\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.461013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.452749 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxpj\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-kube-api-access-frxpj\") pod \"e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.530818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.530697 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:24.569699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.569665 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:24.946020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:24.945094 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:25.105027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:25.096671 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh"] Nov 02 14:04:25.131002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:25.130944 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:25.131148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:25.131053 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:25.131148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:25.131106 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:04:25.139084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:25.139053 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" event={"ID":"df2486b6-a99e-4bee-939e-88263acdd4b7","Type":"ContainerStarted","Data":"4c8f229f6c33aa4caa37013eb4dff0e265cec49c922ff9cfe5541ec30924bae4"} Nov 02 14:04:25.192621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:25.192507 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:25.548370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:25.548290 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:26.148149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:26.148116 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" event={"ID":"df2486b6-a99e-4bee-939e-88263acdd4b7","Type":"ContainerStarted","Data":"215d88483293379475e25fcecf915b2f1d9b1d9efee9b66cde1cd71f1a46db35"} Nov 02 14:04:26.222710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:26.222650 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" podStartSLOduration=2.22263581 podStartE2EDuration="2.22263581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:26.207333762 +0000 UTC m=+1214.932955532" watchObservedRunningTime="2026-11-02 14:04:26.22263581 +0000 UTC m=+1214.948257580" Nov 02 14:04:26.570284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:26.570193 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:26.575431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:26.575405 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:26.939196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:26.931567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:27.154789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:27.154760 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:27.155862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:27.155838 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:04:27.306245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:27.306154 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:28.224820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.224782 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" podUID="99c3140d-c1e2-4d56-9d7c-6090de669c2a" containerName="main" containerID="cri-o://933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816" gracePeriod=45 Nov 02 14:04:28.504971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.504938 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:04:28.591772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.591735 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-dshm\") pod \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " Nov 02 14:04:28.592009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.591832 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c3140d-c1e2-4d56-9d7c-6090de669c2a-kube-api-access-5f5dz\" (UniqueName: \"kubernetes.io/projected/99c3140d-c1e2-4d56-9d7c-6090de669c2a-kube-api-access-5f5dz\") pod \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " Nov 02 14:04:28.592009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.591910 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-model-cache\") pod \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " Nov 02 14:04:28.592009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.591976 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tls-certs\") pod \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " Nov 02 14:04:28.592219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.592153 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tmp-dir\") pod \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " Nov 02 14:04:28.592279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.592223 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-home\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-home\") pod \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\" (UID: \"99c3140d-c1e2-4d56-9d7c-6090de669c2a\") " Nov 02 14:04:28.592897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.592862 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-model-cache" pod "99c3140d-c1e2-4d56-9d7c-6090de669c2a" (UID: "99c3140d-c1e2-4d56-9d7c-6090de669c2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:04:28.593120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.593095 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tmp-dir" pod "99c3140d-c1e2-4d56-9d7c-6090de669c2a" (UID: "99c3140d-c1e2-4d56-9d7c-6090de669c2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:04:28.593405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.593382 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:28.593490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.593411 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:28.593616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.593594 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-home" pod "99c3140d-c1e2-4d56-9d7c-6090de669c2a" (UID: "99c3140d-c1e2-4d56-9d7c-6090de669c2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:04:28.596800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.596774 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-dshm" pod "99c3140d-c1e2-4d56-9d7c-6090de669c2a" (UID: "99c3140d-c1e2-4d56-9d7c-6090de669c2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:04:28.596917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.596881 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tls-certs" pod "99c3140d-c1e2-4d56-9d7c-6090de669c2a" (UID: "99c3140d-c1e2-4d56-9d7c-6090de669c2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:04:28.596994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.596934 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c3140d-c1e2-4d56-9d7c-6090de669c2a-kube-api-access-5f5dz" pod "99c3140d-c1e2-4d56-9d7c-6090de669c2a" (UID: "99c3140d-c1e2-4d56-9d7c-6090de669c2a"). InnerVolumeSpecName "kube-api-access-5f5dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:04:28.695128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.695091 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f5dz\" (UniqueName: \"kubernetes.io/projected/99c3140d-c1e2-4d56-9d7c-6090de669c2a-kube-api-access-5f5dz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:28.695128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.695128 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c3140d-c1e2-4d56-9d7c-6090de669c2a-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:28.695335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.695145 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:28.695335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.695158 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c3140d-c1e2-4d56-9d7c-6090de669c2a-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:28.708245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.708210 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:28.993026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:28.990475 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:29.169273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.169238 5591 generic.go:356] "Generic (PLEG): container finished" podID="99c3140d-c1e2-4d56-9d7c-6090de669c2a" containerID="933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816" exitCode=0 Nov 02 14:04:29.169461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.169287 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" Nov 02 14:04:29.169461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.169322 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" event={"ID":"99c3140d-c1e2-4d56-9d7c-6090de669c2a","Type":"ContainerDied","Data":"933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816"} Nov 02 14:04:29.169461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.169365 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk" event={"ID":"99c3140d-c1e2-4d56-9d7c-6090de669c2a","Type":"ContainerDied","Data":"6c78326b95b062a011b0f9f844ac7cd5f87f28d57530d44abed1afe0cf8af8ba"} Nov 02 14:04:29.169461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.169383 5591 scope.go:122] "RemoveContainer" containerID="933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816" Nov 02 14:04:29.179857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.179842 5591 scope.go:122] "RemoveContainer" containerID="933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816" Nov 02 14:04:29.180145 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:29.180123 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816\": container with ID starting with 933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816 not found: ID does not exist" containerID="933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816" Nov 02 14:04:29.180197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.180155 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816"} err="failed to get container status \"933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816\": rpc error: code = NotFound desc = could not find container \"933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816\": container with ID starting with 933036411f542766ee0e24dd1f7a12323d94191a11f84ed14a75734d67e59816 not found: ID does not exist" Nov 02 14:04:29.228421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.228385 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk"] Nov 02 14:04:29.234843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.234818 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d776122a/e2e-lifecycle-model-d776122a-kserve-64857bbdd5-n6kqk"] Nov 02 14:04:29.431722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:29.431685 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c3140d-c1e2-4d56-9d7c-6090de669c2a" path="/var/lib/kubelet/pods/99c3140d-c1e2-4d56-9d7c-6090de669c2a/volumes" Nov 02 14:04:30.359470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.359429 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:30.474166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.474130 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49"] Nov 02 14:04:30.474765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.474750 5591 cpu_manager.go:414] "containerMap: removing container" podUID="99c3140d-c1e2-4d56-9d7c-6090de669c2a" containerName="main" Nov 02 14:04:30.474834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.474765 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c3140d-c1e2-4d56-9d7c-6090de669c2a" containerName="main" Nov 02 14:04:30.474871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.474864 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c3140d-c1e2-4d56-9d7c-6090de669c2a" containerName="main" Nov 02 14:04:30.478258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.478241 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:30.495929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.495899 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ae366a\"" Nov 02 14:04:30.509133 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.509100 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ae366a-dockercfg-7lk9t\"" Nov 02 14:04:30.512373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.512345 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49"] Nov 02 14:04:30.518716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.518688 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dw2\" (UniqueName: \"kubernetes.io/projected/ade4f8af-bdbe-487d-9860-d733e8150ebb-kube-api-access-s9dw2\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49\" (UID: \"ade4f8af-bdbe-487d-9860-d733e8150ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:30.518857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.518837 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade4f8af-bdbe-487d-9860-d733e8150ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49\" (UID: \"ade4f8af-bdbe-487d-9860-d733e8150ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:30.550514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.550459 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:30.626095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.625994 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dw2\" (UniqueName: \"kubernetes.io/projected/ade4f8af-bdbe-487d-9860-d733e8150ebb-kube-api-access-s9dw2\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49\" (UID: \"ade4f8af-bdbe-487d-9860-d733e8150ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:30.626438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.626129 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade4f8af-bdbe-487d-9860-d733e8150ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49\" (UID: \"ade4f8af-bdbe-487d-9860-d733e8150ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:30.633296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.627171 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade4f8af-bdbe-487d-9860-d733e8150ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49\" (UID: \"ade4f8af-bdbe-487d-9860-d733e8150ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:30.633296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.629652 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8"] Nov 02 14:04:30.644350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.644323 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:30.699420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.699294 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8"] Nov 02 14:04:30.701552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.701491 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dw2\" (UniqueName: \"kubernetes.io/projected/ade4f8af-bdbe-487d-9860-d733e8150ebb-kube-api-access-s9dw2\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49\" (UID: \"ade4f8af-bdbe-487d-9860-d733e8150ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:30.775162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.772961 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8def2b7e-5b1f-4365-83c5-886b95258f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8\" (UID: \"8def2b7e-5b1f-4365-83c5-886b95258f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:30.775162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.773202 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqjc\" (UniqueName: \"kubernetes.io/projected/8def2b7e-5b1f-4365-83c5-886b95258f20-kube-api-access-8kqjc\") pod \"payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8\" (UID: \"8def2b7e-5b1f-4365-83c5-886b95258f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:30.817005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.798311 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:30.858028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.854486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:30.917279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.916008 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqjc\" (UniqueName: \"kubernetes.io/projected/8def2b7e-5b1f-4365-83c5-886b95258f20-kube-api-access-8kqjc\") pod \"payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8\" (UID: \"8def2b7e-5b1f-4365-83c5-886b95258f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:30.917279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.916150 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8def2b7e-5b1f-4365-83c5-886b95258f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8\" (UID: \"8def2b7e-5b1f-4365-83c5-886b95258f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:30.917514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:30.917371 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8def2b7e-5b1f-4365-83c5-886b95258f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8\" (UID: \"8def2b7e-5b1f-4365-83c5-886b95258f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:31.205663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:31.202966 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqjc\" (UniqueName: \"kubernetes.io/projected/8def2b7e-5b1f-4365-83c5-886b95258f20-kube-api-access-8kqjc\") pod \"payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8\" (UID: \"8def2b7e-5b1f-4365-83c5-886b95258f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:31.229851 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:31.229803 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:04:31.299254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:31.299219 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:31.508356 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:04:31.508322 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade4f8af_bdbe_487d_9860_d733e8150ebb.slice/crio-30282db11125a1c463cde6bf1f34fda61f48317f8a45bd316ed656530b98590c WatchSource:0}: Error finding container 30282db11125a1c463cde6bf1f34fda61f48317f8a45bd316ed656530b98590c: Status 404 returned error can't find the container with id 30282db11125a1c463cde6bf1f34fda61f48317f8a45bd316ed656530b98590c Nov 02 14:04:31.508747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:31.508456 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49"] Nov 02 14:04:31.608502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:31.608478 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8"] Nov 02 14:04:31.608962 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:04:31.608935 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8def2b7e_5b1f_4365_83c5_886b95258f20.slice/crio-4fd91ed413a99dcb75fceb7cba6fb32cc5fdbc2e1ca0d619817ea7a87f198a7e WatchSource:0}: Error finding container 4fd91ed413a99dcb75fceb7cba6fb32cc5fdbc2e1ca0d619817ea7a87f198a7e: Status 404 returned error can't find the container with id 4fd91ed413a99dcb75fceb7cba6fb32cc5fdbc2e1ca0d619817ea7a87f198a7e Nov 02 14:04:32.033172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.033138 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:32.203955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.203921 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:32.223546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.223507 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" event={"ID":"8def2b7e-5b1f-4365-83c5-886b95258f20","Type":"ContainerStarted","Data":"e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165"} Nov 02 14:04:32.223722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.223567 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" event={"ID":"8def2b7e-5b1f-4365-83c5-886b95258f20","Type":"ContainerStarted","Data":"4fd91ed413a99dcb75fceb7cba6fb32cc5fdbc2e1ca0d619817ea7a87f198a7e"} Nov 02 14:04:32.223722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.223597 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:32.225306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.225284 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" event={"ID":"ade4f8af-bdbe-487d-9860-d733e8150ebb","Type":"ContainerStarted","Data":"e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf"} Nov 02 14:04:32.225398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.225325 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" event={"ID":"ade4f8af-bdbe-487d-9860-d733e8150ebb","Type":"ContainerStarted","Data":"30282db11125a1c463cde6bf1f34fda61f48317f8a45bd316ed656530b98590c"} Nov 02 14:04:32.225398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.225366 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:32.266611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.266558 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" podStartSLOduration=2.266545286 podStartE2EDuration="2.266545286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:32.265739338 +0000 UTC m=+1220.991361108" watchObservedRunningTime="2026-11-02 14:04:32.266545286 +0000 UTC m=+1220.992167055" Nov 02 14:04:32.311547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.311440 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" podStartSLOduration=2.311423855 podStartE2EDuration="2.311423855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:32.310308527 +0000 UTC m=+1221.035930301" watchObservedRunningTime="2026-11-02 14:04:32.311423855 +0000 UTC m=+1221.037045624" Nov 02 14:04:32.679006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.678138 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:32.913615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:32.913566 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:34.116066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:34.116032 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:04:34.429309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:34.429280 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:34.621969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:34.621939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:34.791174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:34.791091 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:35.092266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:35.089307 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:35.142342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:35.142313 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:04:36.158817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:36.158782 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:37.756010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:37.752804 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:39.319707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:39.319675 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:39.483594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:39.483549 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:41.054201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:41.054123 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:42.792004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:42.791256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:43.234566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:43.234537 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:04:43.234746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:43.234612 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:04:43.410591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:43.410540 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:44.272954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.272912 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx"] Nov 02 14:04:44.277720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.277699 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.306222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.306190 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx"] Nov 02 14:04:44.306392 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.306379 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ae366a-serving-cert\"" Nov 02 14:04:44.374911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.374875 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483df060-32e2-4c41-92a9-5be3dc61c54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.375100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.375047 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-metrics-certs\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.375237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.375224 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krs8b\" (UniqueName: \"kubernetes.io/projected/483df060-32e2-4c41-92a9-5be3dc61c54f-kube-api-access-krs8b\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.375280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.375254 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.483653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.483612 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:44.485312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.485281 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483df060-32e2-4c41-92a9-5be3dc61c54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.497587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.497551 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-metrics-certs\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.497859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.497836 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krs8b\" (UniqueName: \"kubernetes.io/projected/483df060-32e2-4c41-92a9-5be3dc61c54f-kube-api-access-krs8b\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.498798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.498770 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.499123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.491294 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483df060-32e2-4c41-92a9-5be3dc61c54f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.499227 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:44.498912 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-serving-cert: secret "maas-api-e2e-ipp-a-ae366a-serving-cert" not found Nov 02 14:04:44.499294 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:44.499239 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls podName:483df060-32e2-4c41-92a9-5be3dc61c54f nodeName:}" failed. No retries permitted until 2026-11-02 14:04:44.999217717 +0000 UTC m=+1233.724839467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls") pod "maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" (UID: "483df060-32e2-4c41-92a9-5be3dc61c54f") : secret "maas-api-e2e-ipp-a-ae366a-serving-cert" not found Nov 02 14:04:44.508016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.504580 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-metrics-certs\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.553672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.553584 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krs8b\" (UniqueName: \"kubernetes.io/projected/483df060-32e2-4c41-92a9-5be3dc61c54f-kube-api-access-krs8b\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:44.652788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.652726 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:44.821913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:44.821811 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:45.007288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:45.007255 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:45.009731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:45.009707 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:45.203434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:45.203396 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:45.679671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:45.679629 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx"] Nov 02 14:04:46.026924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:46.026897 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:46.542975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:46.542930 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" event={"ID":"483df060-32e2-4c41-92a9-5be3dc61c54f","Type":"ContainerStarted","Data":"8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6"} Nov 02 14:04:46.543764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:46.543738 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" event={"ID":"483df060-32e2-4c41-92a9-5be3dc61c54f","Type":"ContainerStarted","Data":"f4e2cdea11de46d3cd5eb03eb2740a047ca142b0ee98a2c2cb61d5ba953264de"} Nov 02 14:04:46.544439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:46.544421 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:46.627517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:46.627432 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" podStartSLOduration=2.343168812 podStartE2EDuration="2.627413001s" totalImagesPullingTime="284.244189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:46.625755048 +0000 UTC m=+1235.351376819" watchObservedRunningTime="2026-11-02 14:04:46.627413001 +0000 UTC m=+1235.353034773" Nov 02 14:04:46.773010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:46.766866 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:48.426317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:48.425614 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:49.402959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:49.402924 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:49.952238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:49.952203 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm"] Nov 02 14:04:50.079259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:50.079227 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:50.322885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:50.322790 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:50.967008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:50.956618 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:04:52.237317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:52.237266 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:52.477009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:52.455188 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:52.583062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:52.578159 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:04:54.556875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:54.552708 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:54.758319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:54.758276 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb"] Nov 02 14:04:54.769257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:54.769227 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh"] Nov 02 14:04:54.956162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:54.956100 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" podUID="34e21615-83ba-46fa-b632-b225333e56a1" containerName="payload-pre-processing" containerID="cri-o://ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd" gracePeriod=25 Nov 02 14:04:54.959117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:54.959088 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:55.195605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.195528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:55.478354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.478324 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:04:55.616004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.615904 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e21615-83ba-46fa-b632-b225333e56a1-kube-api-access-fr8kj\" (UniqueName: \"kubernetes.io/projected/34e21615-83ba-46fa-b632-b225333e56a1-kube-api-access-fr8kj\") pod \"34e21615-83ba-46fa-b632-b225333e56a1\" (UID: \"34e21615-83ba-46fa-b632-b225333e56a1\") " Nov 02 14:04:55.617056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.617030 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e21615-83ba-46fa-b632-b225333e56a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e21615-83ba-46fa-b632-b225333e56a1-plugins-config-volume\") pod \"34e21615-83ba-46fa-b632-b225333e56a1\" (UID: \"34e21615-83ba-46fa-b632-b225333e56a1\") " Nov 02 14:04:55.619390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.619366 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e21615-83ba-46fa-b632-b225333e56a1-plugins-config-volume" pod "34e21615-83ba-46fa-b632-b225333e56a1" (UID: "34e21615-83ba-46fa-b632-b225333e56a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:04:55.693392 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.693357 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e21615-83ba-46fa-b632-b225333e56a1-kube-api-access-fr8kj" pod "34e21615-83ba-46fa-b632-b225333e56a1" (UID: "34e21615-83ba-46fa-b632-b225333e56a1"). InnerVolumeSpecName "kube-api-access-fr8kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:04:55.752008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.733167 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8kj\" (UniqueName: \"kubernetes.io/projected/34e21615-83ba-46fa-b632-b225333e56a1-kube-api-access-fr8kj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:55.752414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.752378 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e21615-83ba-46fa-b632-b225333e56a1-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:04:55.880466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.876219 5591 generic.go:356] "Generic (PLEG): container finished" podID="34e21615-83ba-46fa-b632-b225333e56a1" containerID="ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd" exitCode=0 Nov 02 14:04:55.880466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.876356 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" event={"ID":"34e21615-83ba-46fa-b632-b225333e56a1","Type":"ContainerDied","Data":"ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd"} Nov 02 14:04:55.880466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.876406 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" event={"ID":"34e21615-83ba-46fa-b632-b225333e56a1","Type":"ContainerDied","Data":"e87a23246be2d1170a065815b2fe5e9f60ac686a67aba5bdbd18497ad09d4a45"} Nov 02 14:04:55.880466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.876438 5591 scope.go:122] "RemoveContainer" containerID="ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd" Nov 02 14:04:55.880466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.876622 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm" Nov 02 14:04:55.896689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.896666 5591 scope.go:122] "RemoveContainer" containerID="ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd" Nov 02 14:04:55.896995 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:04:55.896957 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd\": container with ID starting with ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd not found: ID does not exist" containerID="ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd" Nov 02 14:04:55.897047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.897025 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd"} err="failed to get container status \"ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd\": rpc error: code = NotFound desc = could not find container \"ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd\": container with ID starting with ed5635c90fb42f89ce1166628fbc3e36c389820f370a033c88986ca7c8bf6ddd not found: ID does not exist" Nov 02 14:04:55.963608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.963570 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm"] Nov 02 14:04:55.968631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:55.968601 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-qtvmm"] Nov 02 14:04:56.542018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:56.540400 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:57.459035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:57.437905 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e21615-83ba-46fa-b632-b225333e56a1" path="/var/lib/kubelet/pods/34e21615-83ba-46fa-b632-b225333e56a1/volumes" Nov 02 14:04:58.176002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:58.175477 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:58.374006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:58.373133 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:58.972009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:58.969202 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:59.400441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.400405 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7"] Nov 02 14:04:59.400661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.400638 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" containerID="cri-o://c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819" gracePeriod=30 Nov 02 14:04:59.487823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.487758 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:59.664159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.664036 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:04:59.759818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.759735 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" podUID="6557f55f-06ff-44ee-93fc-0294f1f00317" containerName="payload-processing" containerID="cri-o://9f5ed4302bbc378853850e627567183acd8342c4620920d57d3a34b449655611" gracePeriod=25 Nov 02 14:04:59.769782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.769750 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" podUID="428bbec0-8481-4fa8-a023-40b8ad5325e7" containerName="payload-pre-processing" containerID="cri-o://cd73cc6b36b3a1af0574bfe1af1e1d37e6ae9fdbec1a64cc967a7f9ed2d73476" gracePeriod=25 Nov 02 14:04:59.925093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.918143 5591 generic.go:356] "Generic (PLEG): container finished" podID="6557f55f-06ff-44ee-93fc-0294f1f00317" containerID="9f5ed4302bbc378853850e627567183acd8342c4620920d57d3a34b449655611" exitCode=0 Nov 02 14:04:59.925093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.918323 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" event={"ID":"6557f55f-06ff-44ee-93fc-0294f1f00317","Type":"ContainerDied","Data":"9f5ed4302bbc378853850e627567183acd8342c4620920d57d3a34b449655611"} Nov 02 14:04:59.925093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.919847 5591 generic.go:356] "Generic (PLEG): container finished" podID="428bbec0-8481-4fa8-a023-40b8ad5325e7" containerID="cd73cc6b36b3a1af0574bfe1af1e1d37e6ae9fdbec1a64cc967a7f9ed2d73476" exitCode=0 Nov 02 14:04:59.925093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:04:59.919912 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" event={"ID":"428bbec0-8481-4fa8-a023-40b8ad5325e7","Type":"ContainerDied","Data":"cd73cc6b36b3a1af0574bfe1af1e1d37e6ae9fdbec1a64cc967a7f9ed2d73476"} Nov 02 14:05:00.083960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.083930 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:05:00.129076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.129045 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:05:00.187508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.187408 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428bbec0-8481-4fa8-a023-40b8ad5325e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428bbec0-8481-4fa8-a023-40b8ad5325e7-plugins-config-volume\") pod \"428bbec0-8481-4fa8-a023-40b8ad5325e7\" (UID: \"428bbec0-8481-4fa8-a023-40b8ad5325e7\") " Nov 02 14:05:00.187508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.187477 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6557f55f-06ff-44ee-93fc-0294f1f00317-kube-api-access-jngnt\" (UniqueName: \"kubernetes.io/projected/6557f55f-06ff-44ee-93fc-0294f1f00317-kube-api-access-jngnt\") pod \"6557f55f-06ff-44ee-93fc-0294f1f00317\" (UID: \"6557f55f-06ff-44ee-93fc-0294f1f00317\") " Nov 02 14:05:00.187735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.187603 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6557f55f-06ff-44ee-93fc-0294f1f00317-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6557f55f-06ff-44ee-93fc-0294f1f00317-plugins-config-volume\") pod \"6557f55f-06ff-44ee-93fc-0294f1f00317\" (UID: \"6557f55f-06ff-44ee-93fc-0294f1f00317\") " Nov 02 14:05:00.187735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.187681 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428bbec0-8481-4fa8-a023-40b8ad5325e7-kube-api-access-mnxcr\" (UniqueName: \"kubernetes.io/projected/428bbec0-8481-4fa8-a023-40b8ad5325e7-kube-api-access-mnxcr\") pod \"428bbec0-8481-4fa8-a023-40b8ad5325e7\" (UID: \"428bbec0-8481-4fa8-a023-40b8ad5325e7\") " Nov 02 14:05:00.188092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.188059 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6557f55f-06ff-44ee-93fc-0294f1f00317-plugins-config-volume" pod "6557f55f-06ff-44ee-93fc-0294f1f00317" (UID: "6557f55f-06ff-44ee-93fc-0294f1f00317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:00.188270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.188251 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428bbec0-8481-4fa8-a023-40b8ad5325e7-plugins-config-volume" pod "428bbec0-8481-4fa8-a023-40b8ad5325e7" (UID: "428bbec0-8481-4fa8-a023-40b8ad5325e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:00.188667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.188647 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428bbec0-8481-4fa8-a023-40b8ad5325e7-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:00.188762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.188671 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6557f55f-06ff-44ee-93fc-0294f1f00317-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:00.190760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.190635 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428bbec0-8481-4fa8-a023-40b8ad5325e7-kube-api-access-mnxcr" pod "428bbec0-8481-4fa8-a023-40b8ad5325e7" (UID: "428bbec0-8481-4fa8-a023-40b8ad5325e7"). InnerVolumeSpecName "kube-api-access-mnxcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:00.191969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.191939 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6557f55f-06ff-44ee-93fc-0294f1f00317-kube-api-access-jngnt" pod "6557f55f-06ff-44ee-93fc-0294f1f00317" (UID: "6557f55f-06ff-44ee-93fc-0294f1f00317"). InnerVolumeSpecName "kube-api-access-jngnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:00.231384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.231344 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:00.290207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.290169 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngnt\" (UniqueName: \"kubernetes.io/projected/6557f55f-06ff-44ee-93fc-0294f1f00317-kube-api-access-jngnt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:00.290207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.290201 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnxcr\" (UniqueName: \"kubernetes.io/projected/428bbec0-8481-4fa8-a023-40b8ad5325e7-kube-api-access-mnxcr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:00.417061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.417029 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:00.611693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.611620 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:00.858162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.858131 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:00.930047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.929977 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" Nov 02 14:05:00.930047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.930028 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh" event={"ID":"428bbec0-8481-4fa8-a023-40b8ad5325e7","Type":"ContainerDied","Data":"0b41af5556d22b77eb9a54c6526083b9742e07a8dc295f5c0371cb6b20f6f0ac"} Nov 02 14:05:00.930299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.930097 5591 scope.go:122] "RemoveContainer" containerID="cd73cc6b36b3a1af0574bfe1af1e1d37e6ae9fdbec1a64cc967a7f9ed2d73476" Nov 02 14:05:00.931526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.931508 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" Nov 02 14:05:00.931586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.931526 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb" event={"ID":"6557f55f-06ff-44ee-93fc-0294f1f00317","Type":"ContainerDied","Data":"0fe3b3d3b86d37c722f0b1b9c4e692274ef4934465301357da1dfd4c4f3b1c75"} Nov 02 14:05:00.942089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:00.942058 5591 scope.go:122] "RemoveContainer" containerID="9f5ed4302bbc378853850e627567183acd8342c4620920d57d3a34b449655611" Nov 02 14:05:01.012184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.012154 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh"] Nov 02 14:05:01.025124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.025094 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-5f4d457b99-lp6xh"] Nov 02 14:05:01.054633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.054600 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb"] Nov 02 14:05:01.062328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.062302 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-57db5d8785-qk2cb"] Nov 02 14:05:01.081129 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:05:01.081098 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:05:01.132909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.132877 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx"] Nov 02 14:05:01.133523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133507 5591 cpu_manager.go:414] "containerMap: removing container" podUID="428bbec0-8481-4fa8-a023-40b8ad5325e7" containerName="payload-pre-processing" Nov 02 14:05:01.133575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133524 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428bbec0-8481-4fa8-a023-40b8ad5325e7" containerName="payload-pre-processing" Nov 02 14:05:01.133575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133551 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6557f55f-06ff-44ee-93fc-0294f1f00317" containerName="payload-processing" Nov 02 14:05:01.133575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133557 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6557f55f-06ff-44ee-93fc-0294f1f00317" containerName="payload-processing" Nov 02 14:05:01.133575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133573 5591 cpu_manager.go:414] "containerMap: removing container" podUID="34e21615-83ba-46fa-b632-b225333e56a1" containerName="payload-pre-processing" Nov 02 14:05:01.133699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133579 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e21615-83ba-46fa-b632-b225333e56a1" containerName="payload-pre-processing" Nov 02 14:05:01.133699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133676 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6557f55f-06ff-44ee-93fc-0294f1f00317" containerName="payload-processing" Nov 02 14:05:01.133699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133687 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e21615-83ba-46fa-b632-b225333e56a1" containerName="payload-pre-processing" Nov 02 14:05:01.133791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.133699 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="428bbec0-8481-4fa8-a023-40b8ad5325e7" containerName="payload-pre-processing" Nov 02 14:05:01.138403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.138383 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.165256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.165214 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-17a7ba-openshift-default-dockercfg-zgv9s\"" Nov 02 14:05:01.173524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.170933 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx"] Nov 02 14:05:01.212240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212151 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7833de90-13be-4b90-a950-1d3083a72f92-istiod-ca-cert\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.212380 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212271 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-certs\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.212380 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212375 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-credential-socket\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.212473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212406 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7833de90-13be-4b90-a950-1d3083a72f92-istio-podinfo\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.212473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212442 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-data\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.212579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212497 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-envoy\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.212579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212522 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-socket\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.212579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212547 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5dw2\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-kube-api-access-l5dw2\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.212724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.212602 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-istio-token\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320399 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-certs\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320561 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-credential-socket\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320596 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7833de90-13be-4b90-a950-1d3083a72f92-istio-podinfo\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320631 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-data\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320686 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-envoy\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320709 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-socket\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320733 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5dw2\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-kube-api-access-l5dw2\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320785 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-istio-token\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.320885 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7833de90-13be-4b90-a950-1d3083a72f92-istiod-ca-cert\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.321449 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-certs\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.322203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.321691 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-credential-socket\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.323255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.323229 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-socket\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.323633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.323613 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-data\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.340943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.340907 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7833de90-13be-4b90-a950-1d3083a72f92-istiod-ca-cert\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.370005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.366097 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-envoy\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.383003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.379544 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-istio-token\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.386828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.386758 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5dw2\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-kube-api-access-l5dw2\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.401478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.399722 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7833de90-13be-4b90-a950-1d3083a72f92-istio-podinfo\") pod \"e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.419014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.411968 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:01.461321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.461259 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:01.567564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.567530 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428bbec0-8481-4fa8-a023-40b8ad5325e7" path="/var/lib/kubelet/pods/428bbec0-8481-4fa8-a023-40b8ad5325e7/volumes" Nov 02 14:05:01.568424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.568406 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6557f55f-06ff-44ee-93fc-0294f1f00317" path="/var/lib/kubelet/pods/6557f55f-06ff-44ee-93fc-0294f1f00317/volumes" Nov 02 14:05:01.590726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.590694 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:01.615457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.613048 5591 patch_prober.go:29] interesting pod/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:05:01.615457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.613106 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:05:01.900416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.900389 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx"] Nov 02 14:05:01.901842 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:01.901812 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7833de90_13be_4b90_a950_1d3083a72f92.slice/crio-0c9cedbf67ae9c0c5f8a22cc7ea2c1ee2d285475ec2ff2c83bd7d15389e77011 WatchSource:0}: Error finding container 0c9cedbf67ae9c0c5f8a22cc7ea2c1ee2d285475ec2ff2c83bd7d15389e77011: Status 404 returned error can't find the container with id 0c9cedbf67ae9c0c5f8a22cc7ea2c1ee2d285475ec2ff2c83bd7d15389e77011 Nov 02 14:05:01.904604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.904560 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:01.904701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.904659 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:01.904763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.904707 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:01.940955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:01.940918 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" event={"ID":"7833de90-13be-4b90-a950-1d3083a72f92","Type":"ContainerStarted","Data":"0c9cedbf67ae9c0c5f8a22cc7ea2c1ee2d285475ec2ff2c83bd7d15389e77011"} Nov 02 14:05:02.318727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.318489 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:02.854235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.854200 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 14:05:02.854699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.854249 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 14:05:02.950731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.950658 5591 generic.go:356] "Generic (PLEG): container finished" podID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerID="cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0" exitCode=0 Nov 02 14:05:02.950882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.950734 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" event={"ID":"33351e87-01da-44ee-98c8-e59f6e0aa709","Type":"ContainerDied","Data":"cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0"} Nov 02 14:05:02.951328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.951308 5591 scope.go:122] "RemoveContainer" containerID="cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0" Nov 02 14:05:02.952682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.952642 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" event={"ID":"7833de90-13be-4b90-a950-1d3083a72f92","Type":"ContainerStarted","Data":"979b5222ffb680c54a7d289f7bf416038308347e89358165da85caecac355d7b"} Nov 02 14:05:02.952948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.952925 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:02.953018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.953004 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:02.953069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:02.953057 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:03.071049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:03.070945 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podStartSLOduration=2.070926993 podStartE2EDuration="2.070926993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:03.067743478 +0000 UTC m=+1251.793365248" watchObservedRunningTime="2026-11-02 14:05:03.070926993 +0000 UTC m=+1251.796548770" Nov 02 14:05:03.466258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:03.466227 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:03.471754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:03.471727 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:03.981008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:03.976516 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" event={"ID":"33351e87-01da-44ee-98c8-e59f6e0aa709","Type":"ContainerStarted","Data":"5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816"} Nov 02 14:05:03.981008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:03.976596 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:04.012017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.011391 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:04.074007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.071118 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:04.630737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.630702 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:05:04.631225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.630910 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 14:05:04.631225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.630975 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 14:05:04.728632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.728607 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809386 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-token\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-token\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809445 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-podinfo\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809551 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-envoy\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809598 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-data\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809658 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-certs\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809699 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-credential-socket\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809797 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-kube-api-access-tt8d6\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-kube-api-access-tt8d6\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809837 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b339ca8-336c-4696-a390-64c9a1ba582a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b339ca8-336c-4696-a390-64c9a1ba582a-istiod-ca-cert\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.809965 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-socket\") pod \"0b339ca8-336c-4696-a390-64c9a1ba582a\" (UID: \"0b339ca8-336c-4696-a390-64c9a1ba582a\") " Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.817409 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-certs" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.817539 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b339ca8-336c-4696-a390-64c9a1ba582a-istiod-ca-cert" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.824084 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-credential-socket" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:04.839004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.825676 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-data" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:04.868561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.843657 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-kube-api-access-tt8d6" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "kube-api-access-tt8d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:04.868561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.845648 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-token" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:04.868561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.846843 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-podinfo" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:05:04.883005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.874115 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-envoy" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:04.883005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.874181 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-socket" pod "0b339ca8-336c-4696-a390-64c9a1ba582a" (UID: "0b339ca8-336c-4696-a390-64c9a1ba582a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931029 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931073 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931089 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931104 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931120 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8d6\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-kube-api-access-tt8d6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931134 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b339ca8-336c-4696-a390-64c9a1ba582a-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931148 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b339ca8-336c-4696-a390-64c9a1ba582a-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931161 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:04.938565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:04.931176 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b339ca8-336c-4696-a390-64c9a1ba582a-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:05.222007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.221365 5591 generic.go:356] "Generic (PLEG): container finished" podID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerID="c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819" exitCode=0 Nov 02 14:05:05.247008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.222492 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" Nov 02 14:05:05.247008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.237249 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" event={"ID":"0b339ca8-336c-4696-a390-64c9a1ba582a","Type":"ContainerDied","Data":"c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819"} Nov 02 14:05:05.247008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.237332 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7" event={"ID":"0b339ca8-336c-4696-a390-64c9a1ba582a","Type":"ContainerDied","Data":"c119dd7115fef31a75f956fbc052d11e626adea032165a3f4a9dede789252552"} Nov 02 14:05:05.247008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.237367 5591 scope.go:122] "RemoveContainer" containerID="c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819" Nov 02 14:05:05.402409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.400601 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7"] Nov 02 14:05:05.410481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.410444 5591 scope.go:122] "RemoveContainer" containerID="1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363" Nov 02 14:05:05.497628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.497598 5591 scope.go:122] "RemoveContainer" containerID="c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819" Nov 02 14:05:05.498167 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:05:05.497978 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819\": container with ID starting with c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819 not found: ID does not exist" containerID="c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819" Nov 02 14:05:05.498167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.498034 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819"} err="failed to get container status \"c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819\": rpc error: code = NotFound desc = could not find container \"c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819\": container with ID starting with c4f31fe25e1a55acb46ae53cc48018e4254122ab6893490feb40a4cac01b9819 not found: ID does not exist" Nov 02 14:05:05.498167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.498060 5591 scope.go:122] "RemoveContainer" containerID="1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363" Nov 02 14:05:05.498430 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:05:05.498360 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363\": container with ID starting with 1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363 not found: ID does not exist" containerID="1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363" Nov 02 14:05:05.498430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.498392 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363"} err="failed to get container status \"1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363\": rpc error: code = NotFound desc = could not find container \"1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363\": container with ID starting with 1b36c401ec729aca991a0449ef140f64027de927a5ea0b019cb00e29af883363 not found: ID does not exist" Nov 02 14:05:05.498736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.498717 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8d5eec-openshift-default-687d865b9f-szpd7"] Nov 02 14:05:05.640018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.639664 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 14:05:05.640018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.639822 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 14:05:05.658043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.658008 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:05.979010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:05.969796 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:06.631510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:06.631479 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 14:05:06.631800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:06.631754 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 14:05:06.947061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:06.946592 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:07.449009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.448009 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" path="/var/lib/kubelet/pods/0b339ca8-336c-4696-a390-64c9a1ba582a/volumes" Nov 02 14:05:07.639341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.639089 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 14:05:07.639341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.639166 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 14:05:07.770061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.767947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:07.890116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.880220 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb"] Nov 02 14:05:07.890116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.881359 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" Nov 02 14:05:07.890116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.881382 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" Nov 02 14:05:07.890116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.881403 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" Nov 02 14:05:07.890116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.881411 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" Nov 02 14:05:07.890116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.881581 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" Nov 02 14:05:07.890116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.881597 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b339ca8-336c-4696-a390-64c9a1ba582a" containerName="istio-proxy" Nov 02 14:05:07.890116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.886274 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:07.896006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.895680 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6"] Nov 02 14:05:07.899302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.899283 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:07.914782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.914757 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-17a7ba\"" Nov 02 14:05:07.916099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.916077 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb"] Nov 02 14:05:07.920553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:07.920528 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6"] Nov 02 14:05:08.182792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.182756 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-17a7ba-dockercfg-rgrh8\"" Nov 02 14:05:08.245008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.204368 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:08.290835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.209309 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2bbh\" (UniqueName: \"kubernetes.io/projected/2e904d1f-b4b5-42b1-82c2-5c0272d73582-kube-api-access-w2bbh\") pod \"payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb\" (UID: \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:08.290835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.288383 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd7fe58-f59f-43f3-a31a-b45bab58a160-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6\" (UID: \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:08.290835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.288436 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8sms\" (UniqueName: \"kubernetes.io/projected/0bd7fe58-f59f-43f3-a31a-b45bab58a160-kube-api-access-p8sms\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6\" (UID: \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:08.290835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.288620 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e904d1f-b4b5-42b1-82c2-5c0272d73582-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb\" (UID: \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:08.390561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.390522 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2bbh\" (UniqueName: \"kubernetes.io/projected/2e904d1f-b4b5-42b1-82c2-5c0272d73582-kube-api-access-w2bbh\") pod \"payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb\" (UID: \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:08.390729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.390597 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd7fe58-f59f-43f3-a31a-b45bab58a160-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6\" (UID: \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:08.390729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.390630 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8sms\" (UniqueName: \"kubernetes.io/projected/0bd7fe58-f59f-43f3-a31a-b45bab58a160-kube-api-access-p8sms\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6\" (UID: \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:08.390823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.390738 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e904d1f-b4b5-42b1-82c2-5c0272d73582-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb\" (UID: \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:08.391504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.391483 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd7fe58-f59f-43f3-a31a-b45bab58a160-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6\" (UID: \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:08.391552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.391486 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e904d1f-b4b5-42b1-82c2-5c0272d73582-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb\" (UID: \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:08.424112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.424053 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2bbh\" (UniqueName: \"kubernetes.io/projected/2e904d1f-b4b5-42b1-82c2-5c0272d73582-kube-api-access-w2bbh\") pod \"payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb\" (UID: \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:08.424286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.424263 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8sms\" (UniqueName: \"kubernetes.io/projected/0bd7fe58-f59f-43f3-a31a-b45bab58a160-kube-api-access-p8sms\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6\" (UID: \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:08.464378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.464232 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:08.504563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.504272 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:08.563025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.561263 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:08.695079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.695041 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 14:05:08.695248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.695098 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 14:05:08.726668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:08.726580 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:09.316178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.316143 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6"] Nov 02 14:05:09.316538 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:09.316433 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd7fe58_f59f_43f3_a31a_b45bab58a160.slice/crio-454695292a6aeb75ac391678d0171f2723616041412c0b13cac1558e5c4c10d4 WatchSource:0}: Error finding container 454695292a6aeb75ac391678d0171f2723616041412c0b13cac1558e5c4c10d4: Status 404 returned error can't find the container with id 454695292a6aeb75ac391678d0171f2723616041412c0b13cac1558e5c4c10d4 Nov 02 14:05:09.365104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.364029 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" event={"ID":"0bd7fe58-f59f-43f3-a31a-b45bab58a160","Type":"ContainerStarted","Data":"454695292a6aeb75ac391678d0171f2723616041412c0b13cac1558e5c4c10d4"} Nov 02 14:05:09.365104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.364777 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8"] Nov 02 14:05:09.368235 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:09.368215 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e904d1f_b4b5_42b1_82c2_5c0272d73582.slice/crio-6453d0970e890516c32c254bb4e78eb98683923d81b6536a66d333a73f756545 WatchSource:0}: Error finding container 6453d0970e890516c32c254bb4e78eb98683923d81b6536a66d333a73f756545: Status 404 returned error can't find the container with id 6453d0970e890516c32c254bb4e78eb98683923d81b6536a66d333a73f756545 Nov 02 14:05:09.377704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.377683 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb"] Nov 02 14:05:09.380942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.380916 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49"] Nov 02 14:05:09.475951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.475845 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:09.647759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.647145 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 14:05:09.647759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.647217 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 14:05:09.650252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:09.650220 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:10.376307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:10.376267 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" event={"ID":"2e904d1f-b4b5-42b1-82c2-5c0272d73582","Type":"ContainerStarted","Data":"c9a0bafddbf4114738bea69c14066c47cc2cab3fa16b553be849804a9400cf52"} Nov 02 14:05:10.377211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:10.377183 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" event={"ID":"2e904d1f-b4b5-42b1-82c2-5c0272d73582","Type":"ContainerStarted","Data":"6453d0970e890516c32c254bb4e78eb98683923d81b6536a66d333a73f756545"} Nov 02 14:05:10.378857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:10.378820 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:10.379950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:10.379924 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" event={"ID":"0bd7fe58-f59f-43f3-a31a-b45bab58a160","Type":"ContainerStarted","Data":"31793f3d99aaf58235dc3d5bf573ea5502507863a678092df178ac52ba902cc0"} Nov 02 14:05:10.380897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:10.380878 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:10.424217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:10.424168 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" podStartSLOduration=3.424153629 podStartE2EDuration="3.424153629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:10.42180176 +0000 UTC m=+1259.147423530" watchObservedRunningTime="2026-11-02 14:05:10.424153629 +0000 UTC m=+1259.149775399" Nov 02 14:05:10.641040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:10.637598 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 14:05:10.641040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:10.637661 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 14:05:11.464008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:11.453079 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:11.664069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:11.664030 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:05:11.665163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:11.665139 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:05:11.667671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:11.667645 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:05:11.728113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:11.728017 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" podStartSLOduration=4.728004694 podStartE2EDuration="4.728004694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:10.468429966 +0000 UTC m=+1259.194051746" watchObservedRunningTime="2026-11-02 14:05:11.728004694 +0000 UTC m=+1260.453626464" Nov 02 14:05:13.170003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:13.169470 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:14.365568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.365520 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" podUID="8def2b7e-5b1f-4365-83c5-886b95258f20" containerName="payload-processing" containerID="cri-o://e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165" gracePeriod=25 Nov 02 14:05:14.382253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.382220 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" podUID="ade4f8af-bdbe-487d-9860-d733e8150ebb" containerName="payload-pre-processing" containerID="cri-o://e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf" gracePeriod=25 Nov 02 14:05:14.461429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.461391 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:14.623647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.623557 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:14.693000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.692954 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:05:14.774935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.774911 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:05:14.798955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.798925 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8def2b7e-5b1f-4365-83c5-886b95258f20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8def2b7e-5b1f-4365-83c5-886b95258f20-plugins-config-volume\") pod \"8def2b7e-5b1f-4365-83c5-886b95258f20\" (UID: \"8def2b7e-5b1f-4365-83c5-886b95258f20\") " Nov 02 14:05:14.799182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.799163 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8def2b7e-5b1f-4365-83c5-886b95258f20-kube-api-access-8kqjc\" (UniqueName: \"kubernetes.io/projected/8def2b7e-5b1f-4365-83c5-886b95258f20-kube-api-access-8kqjc\") pod \"8def2b7e-5b1f-4365-83c5-886b95258f20\" (UID: \"8def2b7e-5b1f-4365-83c5-886b95258f20\") " Nov 02 14:05:14.799361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.799341 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8def2b7e-5b1f-4365-83c5-886b95258f20-plugins-config-volume" pod "8def2b7e-5b1f-4365-83c5-886b95258f20" (UID: "8def2b7e-5b1f-4365-83c5-886b95258f20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:14.799755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.799733 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8def2b7e-5b1f-4365-83c5-886b95258f20-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:14.801213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.801189 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8def2b7e-5b1f-4365-83c5-886b95258f20-kube-api-access-8kqjc" pod "8def2b7e-5b1f-4365-83c5-886b95258f20" (UID: "8def2b7e-5b1f-4365-83c5-886b95258f20"). InnerVolumeSpecName "kube-api-access-8kqjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:14.901000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.900947 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade4f8af-bdbe-487d-9860-d733e8150ebb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade4f8af-bdbe-487d-9860-d733e8150ebb-plugins-config-volume\") pod \"ade4f8af-bdbe-487d-9860-d733e8150ebb\" (UID: \"ade4f8af-bdbe-487d-9860-d733e8150ebb\") " Nov 02 14:05:14.901172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.901025 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade4f8af-bdbe-487d-9860-d733e8150ebb-kube-api-access-s9dw2\" (UniqueName: \"kubernetes.io/projected/ade4f8af-bdbe-487d-9860-d733e8150ebb-kube-api-access-s9dw2\") pod \"ade4f8af-bdbe-487d-9860-d733e8150ebb\" (UID: \"ade4f8af-bdbe-487d-9860-d733e8150ebb\") " Nov 02 14:05:14.901444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.901421 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade4f8af-bdbe-487d-9860-d733e8150ebb-plugins-config-volume" pod "ade4f8af-bdbe-487d-9860-d733e8150ebb" (UID: "ade4f8af-bdbe-487d-9860-d733e8150ebb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:14.901872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.901850 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade4f8af-bdbe-487d-9860-d733e8150ebb-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:14.901960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.901873 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kqjc\" (UniqueName: \"kubernetes.io/projected/8def2b7e-5b1f-4365-83c5-886b95258f20-kube-api-access-8kqjc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:14.903301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:14.903277 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade4f8af-bdbe-487d-9860-d733e8150ebb-kube-api-access-s9dw2" pod "ade4f8af-bdbe-487d-9860-d733e8150ebb" (UID: "ade4f8af-bdbe-487d-9860-d733e8150ebb"). InnerVolumeSpecName "kube-api-access-s9dw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:15.003248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.003211 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dw2\" (UniqueName: \"kubernetes.io/projected/ade4f8af-bdbe-487d-9860-d733e8150ebb-kube-api-access-s9dw2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:15.054009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.053076 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:15.414088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.414052 5591 generic.go:356] "Generic (PLEG): container finished" podID="8def2b7e-5b1f-4365-83c5-886b95258f20" containerID="e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165" exitCode=0 Nov 02 14:05:15.414531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.414093 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" Nov 02 14:05:15.414531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.414130 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" event={"ID":"8def2b7e-5b1f-4365-83c5-886b95258f20","Type":"ContainerDied","Data":"e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165"} Nov 02 14:05:15.414531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.414206 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8" event={"ID":"8def2b7e-5b1f-4365-83c5-886b95258f20","Type":"ContainerDied","Data":"4fd91ed413a99dcb75fceb7cba6fb32cc5fdbc2e1ca0d619817ea7a87f198a7e"} Nov 02 14:05:15.414531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.414246 5591 scope.go:122] "RemoveContainer" containerID="e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165" Nov 02 14:05:15.416200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.416177 5591 generic.go:356] "Generic (PLEG): container finished" podID="ade4f8af-bdbe-487d-9860-d733e8150ebb" containerID="e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf" exitCode=0 Nov 02 14:05:15.416315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.416294 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" event={"ID":"ade4f8af-bdbe-487d-9860-d733e8150ebb","Type":"ContainerDied","Data":"e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf"} Nov 02 14:05:15.416361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.416341 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" event={"ID":"ade4f8af-bdbe-487d-9860-d733e8150ebb","Type":"ContainerDied","Data":"30282db11125a1c463cde6bf1f34fda61f48317f8a45bd316ed656530b98590c"} Nov 02 14:05:15.416434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.416422 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49" Nov 02 14:05:15.425628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.425579 5591 scope.go:122] "RemoveContainer" containerID="e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165" Nov 02 14:05:15.428333 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:05:15.428025 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165\": container with ID starting with e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165 not found: ID does not exist" containerID="e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165" Nov 02 14:05:15.428333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.428062 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165"} err="failed to get container status \"e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165\": rpc error: code = NotFound desc = could not find container \"e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165\": container with ID starting with e39e34ca9d5c1c59248824ad3ee1ee93a847d1db2a79a1e7ad5e90d718f5d165 not found: ID does not exist" Nov 02 14:05:15.428333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.428103 5591 scope.go:122] "RemoveContainer" containerID="e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf" Nov 02 14:05:15.461162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.460720 5591 scope.go:122] "RemoveContainer" containerID="e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf" Nov 02 14:05:15.461808 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:05:15.461773 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf\": container with ID starting with e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf not found: ID does not exist" containerID="e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf" Nov 02 14:05:15.461948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.461820 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf"} err="failed to get container status \"e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf\": rpc error: code = NotFound desc = could not find container \"e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf\": container with ID starting with e8aa5472a14f94187200c42c76d7287ad5abab8487fc671fc64a311a616229cf not found: ID does not exist" Nov 02 14:05:15.580259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.580222 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:15.584480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.584453 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8"] Nov 02 14:05:15.588261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.588239 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-9cd4f967c-qkpl8"] Nov 02 14:05:15.608141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.608116 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49"] Nov 02 14:05:15.616657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.616630 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-56d46cc6b7-8hf49"] Nov 02 14:05:15.751068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:15.750438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:16.230039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:16.224627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:16.492439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:16.492361 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx"] Nov 02 14:05:16.492887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:16.492619 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" containerID="cri-o://0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f" gracePeriod=30 Nov 02 14:05:16.573720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:16.573672 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:16.754426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:16.754328 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:05:17.167684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.167647 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:17.377531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.377490 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:17.433636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.433546 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8def2b7e-5b1f-4365-83c5-886b95258f20" path="/var/lib/kubelet/pods/8def2b7e-5b1f-4365-83c5-886b95258f20/volumes" Nov 02 14:05:17.434174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.434149 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade4f8af-bdbe-487d-9860-d733e8150ebb" path="/var/lib/kubelet/pods/ade4f8af-bdbe-487d-9860-d733e8150ebb/volumes" Nov 02 14:05:17.828092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.823664 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc"] Nov 02 14:05:17.828092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.824677 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8def2b7e-5b1f-4365-83c5-886b95258f20" containerName="payload-processing" Nov 02 14:05:17.828092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.824702 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8def2b7e-5b1f-4365-83c5-886b95258f20" containerName="payload-processing" Nov 02 14:05:17.828092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.824784 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ade4f8af-bdbe-487d-9860-d733e8150ebb" containerName="payload-pre-processing" Nov 02 14:05:17.828092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.824793 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade4f8af-bdbe-487d-9860-d733e8150ebb" containerName="payload-pre-processing" Nov 02 14:05:17.828092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.824922 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade4f8af-bdbe-487d-9860-d733e8150ebb" containerName="payload-pre-processing" Nov 02 14:05:17.828092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.824937 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8def2b7e-5b1f-4365-83c5-886b95258f20" containerName="payload-processing" Nov 02 14:05:17.866010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.865306 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc"] Nov 02 14:05:17.866010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.865437 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.893014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.886853 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c7d2df-openshift-default-dockercfg-5q9w6\"" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927012 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-certs\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927071 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cb9750-f264-4b5f-8069-8af98fdc5001-istiod-ca-cert\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927099 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-podinfo\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927172 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-socket\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927246 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4h4q\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-kube-api-access-q4h4q\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927295 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-data\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927359 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-token\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927434 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-envoy\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.944332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.927546 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-credential-socket\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:17.970832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:17.970592 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029238 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-envoy\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029369 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-credential-socket\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029511 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-certs\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029538 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cb9750-f264-4b5f-8069-8af98fdc5001-istiod-ca-cert\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029564 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-podinfo\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029634 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-socket\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029712 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4h4q\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-kube-api-access-q4h4q\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029766 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-data\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.029839 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-token\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.030034 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-certs\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.030571 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-credential-socket\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.030812 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-socket\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.030964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.030930 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cb9750-f264-4b5f-8069-8af98fdc5001-istiod-ca-cert\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.031413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.031386 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-data\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.032477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.032447 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-envoy\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.033960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.033938 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-podinfo\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.055154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.055118 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4h4q\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-kube-api-access-q4h4q\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.055881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.055851 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-token\") pod \"e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.146065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.134738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:18.212007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.205431 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:18.597361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.597331 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc"] Nov 02 14:05:18.597532 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:18.597507 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cb9750_f264_4b5f_8069_8af98fdc5001.slice/crio-ad449f23b42dffaf6aefdf0316a92852bfaace53b5ce7148105ffbef91a49e91 WatchSource:0}: Error finding container ad449f23b42dffaf6aefdf0316a92852bfaace53b5ce7148105ffbef91a49e91: Status 404 returned error can't find the container with id ad449f23b42dffaf6aefdf0316a92852bfaace53b5ce7148105ffbef91a49e91 Nov 02 14:05:18.600310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.600283 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:18.600371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.600346 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:18.600407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.600380 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:18.933009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:18.911546 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:19.452067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:19.452024 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" event={"ID":"33cb9750-f264-4b5f-8069-8af98fdc5001","Type":"ContainerStarted","Data":"39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2"} Nov 02 14:05:19.452252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:19.452087 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" event={"ID":"33cb9750-f264-4b5f-8069-8af98fdc5001","Type":"ContainerStarted","Data":"ad449f23b42dffaf6aefdf0316a92852bfaace53b5ce7148105ffbef91a49e91"} Nov 02 14:05:19.536255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:19.536203 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podStartSLOduration=2.536189662 podStartE2EDuration="2.536189662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:19.53437472 +0000 UTC m=+1268.259996491" watchObservedRunningTime="2026-11-02 14:05:19.536189662 +0000 UTC m=+1268.261811432" Nov 02 14:05:19.906613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:19.906547 5591 patch_prober.go:29] interesting pod/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:05:19.906819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:19.906621 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:05:20.205953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.205873 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:20.212519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.212484 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:20.372629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.347778 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:20.498335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.498025 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:20.503322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.500164 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:05:20.565588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.565554 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:20.812195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.812077 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg"] Nov 02 14:05:20.821787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.821616 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg"] Nov 02 14:05:20.821787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.821739 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:20.838654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.838624 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ae366a-dockercfg-2dhnb\"" Nov 02 14:05:20.839328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.839304 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ae366a\"" Nov 02 14:05:20.916388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.916351 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49ts\" (UniqueName: \"kubernetes.io/projected/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-kube-api-access-l49ts\") pod \"payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg\" (UID: \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:20.917725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:20.916492 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg\" (UID: \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:21.017771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.017739 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49ts\" (UniqueName: \"kubernetes.io/projected/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-kube-api-access-l49ts\") pod \"payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg\" (UID: \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:21.017919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.017839 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg\" (UID: \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:21.018555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.018536 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg\" (UID: \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:21.042743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.042708 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49ts\" (UniqueName: \"kubernetes.io/projected/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-kube-api-access-l49ts\") pod \"payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg\" (UID: \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:21.151495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.151456 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:21.459149 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:21.459114 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db34dfb_1f13_4549_8dea_2ad2a31e2a04.slice/crio-bf31ba7b33aeb584f2962bbb1141cacdd66f4f984901819cbf30783f6d49fd1f WatchSource:0}: Error finding container bf31ba7b33aeb584f2962bbb1141cacdd66f4f984901819cbf30783f6d49fd1f: Status 404 returned error can't find the container with id bf31ba7b33aeb584f2962bbb1141cacdd66f4f984901819cbf30783f6d49fd1f Nov 02 14:05:21.460122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.460092 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg"] Nov 02 14:05:21.461365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.461349 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:05:21.514047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.513048 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:21.580666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.575477 5591 generic.go:356] "Generic (PLEG): container finished" podID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerID="215d88483293379475e25fcecf915b2f1d9b1d9efee9b66cde1cd71f1a46db35" exitCode=0 Nov 02 14:05:21.580666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.575581 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" event={"ID":"df2486b6-a99e-4bee-939e-88263acdd4b7","Type":"ContainerDied","Data":"215d88483293379475e25fcecf915b2f1d9b1d9efee9b66cde1cd71f1a46db35"} Nov 02 14:05:21.580666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.576173 5591 scope.go:122] "RemoveContainer" containerID="215d88483293379475e25fcecf915b2f1d9b1d9efee9b66cde1cd71f1a46db35" Nov 02 14:05:21.586902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.585790 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:21.586902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.585895 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:21.586902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.585942 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:21.610823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.610792 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" event={"ID":"6db34dfb-1f13-4549-8dea-2ad2a31e2a04","Type":"ContainerStarted","Data":"bf31ba7b33aeb584f2962bbb1141cacdd66f4f984901819cbf30783f6d49fd1f"} Nov 02 14:05:21.809794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.809761 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:05:21.873653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.873622 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-token\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-token\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.873869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.873669 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-socket\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.873869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.873705 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-envoy\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.873869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.873758 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-kube-api-access-b6z84\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-kube-api-access-b6z84\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.873869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.873799 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-credential-socket\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.874120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.873934 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istiod-ca-cert\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.874196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.874176 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-podinfo\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.874309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.874233 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-certs\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.874413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.874306 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-data\") pod \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\" (UID: \"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5\") " Nov 02 14:05:21.875073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.875047 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-data" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:21.875537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.875517 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-credential-socket" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:21.876019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.875962 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-socket" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:21.877506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.877484 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-token" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:21.877907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.877887 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-certs" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:21.878056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.878030 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istiod-ca-cert" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:21.878656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.878630 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-envoy" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:05:21.879863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.879838 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-kube-api-access-b6z84" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "kube-api-access-b6z84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:21.880242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.880208 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-podinfo" pod "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" (UID: "7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:05:21.933460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.933417 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:21.976273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976209 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:21.976273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976247 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:21.976273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976262 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:21.976273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976274 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:21.976273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976287 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:21.976731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976300 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:21.976731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976316 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6z84\" (UniqueName: \"kubernetes.io/projected/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-kube-api-access-b6z84\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:21.976731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976329 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:21.976731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:21.976342 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:22.116756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.116668 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:22.398326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.398293 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:22.398519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.398367 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:22.784124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.782978 5591 generic.go:356] "Generic (PLEG): container finished" podID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerID="0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f" exitCode=0 Nov 02 14:05:22.784124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.783130 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" event={"ID":"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5","Type":"ContainerDied","Data":"0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f"} Nov 02 14:05:22.784124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.783178 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" event={"ID":"7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5","Type":"ContainerDied","Data":"053f58d326d46901037375ce0dc2dcde3ad8941dfbdf42136e09d9aa9c807540"} Nov 02 14:05:22.784124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.783202 5591 scope.go:122] "RemoveContainer" containerID="0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f" Nov 02 14:05:22.784124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.783328 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx" Nov 02 14:05:22.786878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.786846 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" event={"ID":"df2486b6-a99e-4bee-939e-88263acdd4b7","Type":"ContainerStarted","Data":"f75ca4488842778333bebf3c286dcc22202f74e6c9c0b6e6c27b1fa77d0cb0d2"} Nov 02 14:05:22.797402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.797380 5591 scope.go:122] "RemoveContainer" containerID="551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92" Nov 02 14:05:22.808874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.808852 5591 scope.go:122] "RemoveContainer" containerID="0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f" Nov 02 14:05:22.809205 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:05:22.809186 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f\": container with ID starting with 0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f not found: ID does not exist" containerID="0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f" Nov 02 14:05:22.809279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.809212 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f"} err="failed to get container status \"0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f\": rpc error: code = NotFound desc = could not find container \"0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f\": container with ID starting with 0db575e847a111bd0d8f936856d1d2ae50a88c66252bd59b9d1e0f4358fe2b8f not found: ID does not exist" Nov 02 14:05:22.809279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.809229 5591 scope.go:122] "RemoveContainer" containerID="551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92" Nov 02 14:05:22.809551 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:05:22.809534 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92\": container with ID starting with 551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92 not found: ID does not exist" containerID="551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92" Nov 02 14:05:22.809635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.809556 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92"} err="failed to get container status \"551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92\": rpc error: code = NotFound desc = could not find container \"551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92\": container with ID starting with 551b12ca63e9cfecf4702e39646a011105a4208549e2b27dd1a74b95053c0a92 not found: ID does not exist" Nov 02 14:05:22.863837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.863798 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx"] Nov 02 14:05:22.868470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:22.868443 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d776122a-openshift-default-84b8bf469c-6q7tx"] Nov 02 14:05:23.189934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.189899 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m"] Nov 02 14:05:23.190746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.190718 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" Nov 02 14:05:23.190746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.190744 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" Nov 02 14:05:23.190899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.190776 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" Nov 02 14:05:23.190899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.190784 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" Nov 02 14:05:23.190977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.190903 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" Nov 02 14:05:23.194684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.194666 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.213310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.213289 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-17a7ba-serving-cert\"" Nov 02 14:05:23.218917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.218704 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m"] Nov 02 14:05:23.301849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.301816 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfj8z\" (UniqueName: \"kubernetes.io/projected/15fd5322-ed09-4fb3-87b1-e27d578a7af6-kube-api-access-nfj8z\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.302003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.301906 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-metrics-certs\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.302003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.301964 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.302086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.302004 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fd5322-ed09-4fb3-87b1-e27d578a7af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.358197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.358090 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:23.403782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.403747 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.403974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.403797 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fd5322-ed09-4fb3-87b1-e27d578a7af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.403974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.403952 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfj8z\" (UniqueName: \"kubernetes.io/projected/15fd5322-ed09-4fb3-87b1-e27d578a7af6-kube-api-access-nfj8z\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.404121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.404053 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-metrics-certs\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.404545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.404522 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fd5322-ed09-4fb3-87b1-e27d578a7af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.406695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.406544 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.406873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.406853 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-metrics-certs\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.559082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.545375 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:23.559082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.553763 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfj8z\" (UniqueName: \"kubernetes.io/projected/15fd5322-ed09-4fb3-87b1-e27d578a7af6-kube-api-access-nfj8z\") pod \"maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:23.626314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.625588 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 14:05:23.626314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.625685 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 14:05:23.705356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.705314 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" path="/var/lib/kubelet/pods/7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5/volumes" Nov 02 14:05:23.706304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.706281 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:05:23.808097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:23.808059 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:24.164882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.164845 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m"] Nov 02 14:05:24.168045 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:24.168015 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fd5322_ed09_4fb3_87b1_e27d578a7af6.slice/crio-bc6200b42933c380fa006297ada21f3a3c7681153f1ce7176bc8a82eb2e1831c WatchSource:0}: Error finding container bc6200b42933c380fa006297ada21f3a3c7681153f1ce7176bc8a82eb2e1831c: Status 404 returned error can't find the container with id bc6200b42933c380fa006297ada21f3a3c7681153f1ce7176bc8a82eb2e1831c Nov 02 14:05:24.612102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.593060 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 14:05:24.612102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.605722 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:24.618009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.612462 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 14:05:24.673092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.673041 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w"] Nov 02 14:05:24.675073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.675041 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1e13a3-e1c5-481a-b6da-ec915ab7a6f5" containerName="istio-proxy" Nov 02 14:05:24.679646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.679623 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:24.696737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.696703 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w"] Nov 02 14:05:24.698304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.697230 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c7d2df\"" Nov 02 14:05:24.803220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.793965 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:24.998063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.985441 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmsm\" (UniqueName: \"kubernetes.io/projected/0cef97ab-3764-4163-908d-4ef66716b0aa-kube-api-access-jrmsm\") pod \"payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w\" (UID: \"0cef97ab-3764-4163-908d-4ef66716b0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:24.998063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:24.985614 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cef97ab-3764-4163-908d-4ef66716b0aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w\" (UID: \"0cef97ab-3764-4163-908d-4ef66716b0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:25.066055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.041741 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" event={"ID":"6db34dfb-1f13-4549-8dea-2ad2a31e2a04","Type":"ContainerStarted","Data":"7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421"} Nov 02 14:05:25.066055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.052104 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:25.098014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.088141 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cef97ab-3764-4163-908d-4ef66716b0aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w\" (UID: \"0cef97ab-3764-4163-908d-4ef66716b0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:25.098014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.088425 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmsm\" (UniqueName: \"kubernetes.io/projected/0cef97ab-3764-4163-908d-4ef66716b0aa-kube-api-access-jrmsm\") pod \"payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w\" (UID: \"0cef97ab-3764-4163-908d-4ef66716b0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:25.098014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.089628 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cef97ab-3764-4163-908d-4ef66716b0aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w\" (UID: \"0cef97ab-3764-4163-908d-4ef66716b0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:25.098014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.092567 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" event={"ID":"15fd5322-ed09-4fb3-87b1-e27d578a7af6","Type":"ContainerStarted","Data":"bc6200b42933c380fa006297ada21f3a3c7681153f1ce7176bc8a82eb2e1831c"} Nov 02 14:05:25.214778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.207023 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmsm\" (UniqueName: \"kubernetes.io/projected/0cef97ab-3764-4163-908d-4ef66716b0aa-kube-api-access-jrmsm\") pod \"payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w\" (UID: \"0cef97ab-3764-4163-908d-4ef66716b0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:25.214778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.208059 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" podStartSLOduration=2.556657843 podStartE2EDuration="5.208040057s" totalImagesPullingTime="2.651382214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:25.205537584 +0000 UTC m=+1273.931159354" watchObservedRunningTime="2026-11-02 14:05:25.208040057 +0000 UTC m=+1273.933661828" Nov 02 14:05:25.306827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.306688 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:25.320007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.313724 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:25.571377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.570717 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 14:05:25.571377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.570778 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 14:05:25.766514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:25.766481 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w"] Nov 02 14:05:25.767517 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:25.767485 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cef97ab_3764_4163_908d_4ef66716b0aa.slice/crio-5bab9bfd2f9fbfc12126078aa489e505cf32621cba88bb04451d358dee08cfb8 WatchSource:0}: Error finding container 5bab9bfd2f9fbfc12126078aa489e505cf32621cba88bb04451d358dee08cfb8: Status 404 returned error can't find the container with id 5bab9bfd2f9fbfc12126078aa489e505cf32621cba88bb04451d358dee08cfb8 Nov 02 14:05:26.101838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.101743 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" event={"ID":"15fd5322-ed09-4fb3-87b1-e27d578a7af6","Type":"ContainerStarted","Data":"582539ea4babe74343ee51a8e657437a351133af405ab1a9ab0bd956cbc9f7cf"} Nov 02 14:05:26.102325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.101837 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:26.103670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.103584 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" event={"ID":"0cef97ab-3764-4163-908d-4ef66716b0aa","Type":"ContainerStarted","Data":"8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9"} Nov 02 14:05:26.103670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.103633 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" event={"ID":"0cef97ab-3764-4163-908d-4ef66716b0aa","Type":"ContainerStarted","Data":"5bab9bfd2f9fbfc12126078aa489e505cf32621cba88bb04451d358dee08cfb8"} Nov 02 14:05:26.103871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.103717 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:26.166401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.165528 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" podStartSLOduration=2.628053761 podStartE2EDuration="3.16550424s" totalImagesPullingTime="537.450479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:26.152295903 +0000 UTC m=+1274.877917674" watchObservedRunningTime="2026-11-02 14:05:26.16550424 +0000 UTC m=+1274.891126010" Nov 02 14:05:26.211006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.204378 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" podStartSLOduration=2.204354921 podStartE2EDuration="2.204354921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:26.195859682 +0000 UTC m=+1274.921481455" watchObservedRunningTime="2026-11-02 14:05:26.204354921 +0000 UTC m=+1274.929976695" Nov 02 14:05:26.578009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.577331 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 14:05:26.578009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.577389 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 14:05:26.946826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:26.946569 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:27.157042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:27.155095 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:05:27.410957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:27.410918 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:27.587054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:27.575483 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 14:05:27.587054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:27.575543 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 14:05:27.587054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:27.585523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:27.777077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:27.774478 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:28.574568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:28.574528 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:05:28.575649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:28.575621 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:05:29.214417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:29.214377 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:30.884618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:30.884546 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:31.078385 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:05:31.078339 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:05:31.525000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:31.524962 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:31.702328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:31.702291 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:32.115864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:32.115837 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:05:32.871351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:32.871314 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:33.301683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.301646 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:33.432994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.432954 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2"] Nov 02 14:05:33.436243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.436218 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:33.470005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.456622 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2"] Nov 02 14:05:33.470005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.467157 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fb206f-dockercfg-brj27\"" Nov 02 14:05:33.470005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.467156 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fb206f\"" Nov 02 14:05:33.470005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.468939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:33.648060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.565101 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpz7\" (UniqueName: \"kubernetes.io/projected/c7ba4b1c-559d-49de-9796-c489426d2a07-kube-api-access-6cpz7\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2\" (UID: \"c7ba4b1c-559d-49de-9796-c489426d2a07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:33.648060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.602316 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7ba4b1c-559d-49de-9796-c489426d2a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2\" (UID: \"c7ba4b1c-559d-49de-9796-c489426d2a07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:33.676091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.676032 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n"] Nov 02 14:05:33.729787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.729753 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:33.767088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.758857 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7ba4b1c-559d-49de-9796-c489426d2a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2\" (UID: \"c7ba4b1c-559d-49de-9796-c489426d2a07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:33.767088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.761046 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7ba4b1c-559d-49de-9796-c489426d2a07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2\" (UID: \"c7ba4b1c-559d-49de-9796-c489426d2a07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:33.767088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.761392 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpz7\" (UniqueName: \"kubernetes.io/projected/c7ba4b1c-559d-49de-9796-c489426d2a07-kube-api-access-6cpz7\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2\" (UID: \"c7ba4b1c-559d-49de-9796-c489426d2a07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:33.784873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.784840 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n"] Nov 02 14:05:33.838497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.837744 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpz7\" (UniqueName: \"kubernetes.io/projected/c7ba4b1c-559d-49de-9796-c489426d2a07-kube-api-access-6cpz7\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2\" (UID: \"c7ba4b1c-559d-49de-9796-c489426d2a07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:33.862831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.862794 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbpp\" (UniqueName: \"kubernetes.io/projected/0dd01d69-da94-420f-b1ec-ce6236df44c4-kube-api-access-dcbpp\") pod \"payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n\" (UID: \"0dd01d69-da94-420f-b1ec-ce6236df44c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:33.863057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.862842 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd01d69-da94-420f-b1ec-ce6236df44c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n\" (UID: \"0dd01d69-da94-420f-b1ec-ce6236df44c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:33.964893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.964803 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbpp\" (UniqueName: \"kubernetes.io/projected/0dd01d69-da94-420f-b1ec-ce6236df44c4-kube-api-access-dcbpp\") pod \"payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n\" (UID: \"0dd01d69-da94-420f-b1ec-ce6236df44c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:33.964893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.964875 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd01d69-da94-420f-b1ec-ce6236df44c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n\" (UID: \"0dd01d69-da94-420f-b1ec-ce6236df44c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:33.965626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.965602 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd01d69-da94-420f-b1ec-ce6236df44c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n\" (UID: \"0dd01d69-da94-420f-b1ec-ce6236df44c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:33.989343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:33.989313 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbpp\" (UniqueName: \"kubernetes.io/projected/0dd01d69-da94-420f-b1ec-ce6236df44c4-kube-api-access-dcbpp\") pod \"payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n\" (UID: \"0dd01d69-da94-420f-b1ec-ce6236df44c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:34.048672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:34.048637 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:34.126539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:34.126506 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:34.353679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:34.353655 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2"] Nov 02 14:05:34.354464 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:34.354439 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ba4b1c_559d_49de_9796_c489426d2a07.slice/crio-7061bba0f65ad73c57de683f689c2f8d7ccfc79fb62d58c01390573e2680b3b2 WatchSource:0}: Error finding container 7061bba0f65ad73c57de683f689c2f8d7ccfc79fb62d58c01390573e2680b3b2: Status 404 returned error can't find the container with id 7061bba0f65ad73c57de683f689c2f8d7ccfc79fb62d58c01390573e2680b3b2 Nov 02 14:05:34.454353 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:34.454328 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd01d69_da94_420f_b1ec_ce6236df44c4.slice/crio-108f7209d132294b3f6f5be2f3d77f28ff6f5c0aec133972bfe32561a28c84cb WatchSource:0}: Error finding container 108f7209d132294b3f6f5be2f3d77f28ff6f5c0aec133972bfe32561a28c84cb: Status 404 returned error can't find the container with id 108f7209d132294b3f6f5be2f3d77f28ff6f5c0aec133972bfe32561a28c84cb Nov 02 14:05:34.455042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:34.455017 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n"] Nov 02 14:05:34.559008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:34.555647 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:35.168307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:35.168268 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" event={"ID":"c7ba4b1c-559d-49de-9796-c489426d2a07","Type":"ContainerStarted","Data":"188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122"} Nov 02 14:05:35.168522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:35.168325 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" event={"ID":"c7ba4b1c-559d-49de-9796-c489426d2a07","Type":"ContainerStarted","Data":"7061bba0f65ad73c57de683f689c2f8d7ccfc79fb62d58c01390573e2680b3b2"} Nov 02 14:05:35.168522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:35.168356 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:35.169784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:35.169755 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" event={"ID":"0dd01d69-da94-420f-b1ec-ce6236df44c4","Type":"ContainerStarted","Data":"216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c"} Nov 02 14:05:35.169917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:35.169804 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" event={"ID":"0dd01d69-da94-420f-b1ec-ce6236df44c4","Type":"ContainerStarted","Data":"108f7209d132294b3f6f5be2f3d77f28ff6f5c0aec133972bfe32561a28c84cb"} Nov 02 14:05:35.169917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:35.169836 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:35.213537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:35.213489 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" podStartSLOduration=2.213475854 podStartE2EDuration="2.213475854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:35.20983307 +0000 UTC m=+1283.935454840" watchObservedRunningTime="2026-11-02 14:05:35.213475854 +0000 UTC m=+1283.939097624" Nov 02 14:05:35.257174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:35.257124 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" podStartSLOduration=2.257111341 podStartE2EDuration="2.257111341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:35.255474212 +0000 UTC m=+1283.981095964" watchObservedRunningTime="2026-11-02 14:05:35.257111341 +0000 UTC m=+1283.982733110" Nov 02 14:05:36.105728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:36.105696 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:05:36.197906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:36.197871 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:37.112800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:37.112769 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:05:38.465011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:38.453556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:38.715866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:38.715774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:40.175494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:40.165077 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:05:40.353012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:40.349783 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:41.228010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:41.223536 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:41.702146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:41.702112 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:41.925007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:41.917945 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:42.279040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:42.279003 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:44.120005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:44.115386 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:45.711288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:45.711240 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:45.892403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:45.892364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:46.063365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:46.063272 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb"] Nov 02 14:05:46.102372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:46.102338 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6"] Nov 02 14:05:46.181134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:46.181103 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:05:46.181449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:46.181428 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:05:46.280094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:46.280051 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:46.591311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:46.591271 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:46.804904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:46.804863 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:47.665904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.665866 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:47.784030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.783996 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6"] Nov 02 14:05:47.788667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.788646 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:47.804374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.804346 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60ad62\"" Nov 02 14:05:47.817579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.817546 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60ad62-dockercfg-rvsls\"" Nov 02 14:05:47.821360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.821335 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6"] Nov 02 14:05:47.833609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.833581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:47.954952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.954861 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8"] Nov 02 14:05:47.966749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:47.966720 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:48.048908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.048873 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-kube-api-access-whmzd\") pod \"payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8\" (UID: \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:48.085354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.085308 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cplbc\" (UniqueName: \"kubernetes.io/projected/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-kube-api-access-cplbc\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6\" (UID: \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:48.085772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.085742 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6\" (UID: \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:48.098129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.098090 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8\" (UID: \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:48.108327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.108287 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8"] Nov 02 14:05:48.203867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.203575 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-kube-api-access-whmzd\") pod \"payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8\" (UID: \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:48.203867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.203639 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cplbc\" (UniqueName: \"kubernetes.io/projected/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-kube-api-access-cplbc\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6\" (UID: \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:48.214074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.208761 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6\" (UID: \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:48.214074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.203793 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6\" (UID: \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:48.214074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.209596 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8\" (UID: \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:48.214074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.210665 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8\" (UID: \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:48.228845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.228811 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cplbc\" (UniqueName: \"kubernetes.io/projected/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-kube-api-access-cplbc\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6\" (UID: \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:48.229002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.228861 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-kube-api-access-whmzd\") pod \"payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8\" (UID: \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:48.314393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.314357 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:48.417319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.417287 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:48.563415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.559925 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:48.666007 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:48.665964 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5d3ea4_0b9c_4fec_adb7_b06328b20f8b.slice/crio-c3a117e119d0ac9528e8354a3667361abb536b2b6ec396c24490d499fa785844 WatchSource:0}: Error finding container c3a117e119d0ac9528e8354a3667361abb536b2b6ec396c24490d499fa785844: Status 404 returned error can't find the container with id c3a117e119d0ac9528e8354a3667361abb536b2b6ec396c24490d499fa785844 Nov 02 14:05:48.666375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.666350 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8"] Nov 02 14:05:48.735616 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:48.735568 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7af67e4_3142_44e1_95cc_b69cbd3d46f1.slice/crio-a8ce82cf38fc0b869bc1dd50da5522284355f982258b0f1bd7d63d85d5eb62a7 WatchSource:0}: Error finding container a8ce82cf38fc0b869bc1dd50da5522284355f982258b0f1bd7d63d85d5eb62a7: Status 404 returned error can't find the container with id a8ce82cf38fc0b869bc1dd50da5522284355f982258b0f1bd7d63d85d5eb62a7 Nov 02 14:05:48.736481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:48.736284 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6"] Nov 02 14:05:49.344720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:49.344683 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" event={"ID":"c7af67e4-3142-44e1-95cc-b69cbd3d46f1","Type":"ContainerStarted","Data":"fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d"} Nov 02 14:05:49.345259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:49.344744 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" event={"ID":"c7af67e4-3142-44e1-95cc-b69cbd3d46f1","Type":"ContainerStarted","Data":"a8ce82cf38fc0b869bc1dd50da5522284355f982258b0f1bd7d63d85d5eb62a7"} Nov 02 14:05:49.345259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:49.344782 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:05:49.346706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:49.346671 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" event={"ID":"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b","Type":"ContainerStarted","Data":"76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5"} Nov 02 14:05:49.346855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:49.346738 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:05:49.346855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:49.346766 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" event={"ID":"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b","Type":"ContainerStarted","Data":"c3a117e119d0ac9528e8354a3667361abb536b2b6ec396c24490d499fa785844"} Nov 02 14:05:49.407686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:49.407619 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" podStartSLOduration=2.407602315 podStartE2EDuration="2.407602315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:49.405718996 +0000 UTC m=+1298.131340767" watchObservedRunningTime="2026-11-02 14:05:49.407602315 +0000 UTC m=+1298.133224085" Nov 02 14:05:49.450717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:49.450663 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" podStartSLOduration=2.450648648 podStartE2EDuration="2.450648648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:49.44785087 +0000 UTC m=+1298.173472639" watchObservedRunningTime="2026-11-02 14:05:49.450648648 +0000 UTC m=+1298.176270418" Nov 02 14:05:50.018152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:50.018110 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:05:50.018355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:50.018185 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:05:50.218792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:50.218758 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:50.429170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:50.429138 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:51.064134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.064082 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" podUID="2e904d1f-b4b5-42b1-82c2-5c0272d73582" containerName="payload-processing" containerID="cri-o://c9a0bafddbf4114738bea69c14066c47cc2cab3fa16b553be849804a9400cf52" gracePeriod=25 Nov 02 14:05:51.102772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.102733 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" podUID="0bd7fe58-f59f-43f3-a31a-b45bab58a160" containerName="payload-pre-processing" containerID="cri-o://31793f3d99aaf58235dc3d5bf573ea5502507863a678092df178ac52ba902cc0" gracePeriod=25 Nov 02 14:05:51.363060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.356603 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:51.547737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.536053 5591 generic.go:356] "Generic (PLEG): container finished" podID="0bd7fe58-f59f-43f3-a31a-b45bab58a160" containerID="31793f3d99aaf58235dc3d5bf573ea5502507863a678092df178ac52ba902cc0" exitCode=0 Nov 02 14:05:51.547737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.543771 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" event={"ID":"0bd7fe58-f59f-43f3-a31a-b45bab58a160","Type":"ContainerDied","Data":"31793f3d99aaf58235dc3d5bf573ea5502507863a678092df178ac52ba902cc0"} Nov 02 14:05:51.564024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.555238 5591 generic.go:356] "Generic (PLEG): container finished" podID="2e904d1f-b4b5-42b1-82c2-5c0272d73582" containerID="c9a0bafddbf4114738bea69c14066c47cc2cab3fa16b553be849804a9400cf52" exitCode=0 Nov 02 14:05:51.564024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.555300 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" event={"ID":"2e904d1f-b4b5-42b1-82c2-5c0272d73582","Type":"ContainerDied","Data":"c9a0bafddbf4114738bea69c14066c47cc2cab3fa16b553be849804a9400cf52"} Nov 02 14:05:51.585615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.585586 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:51.620244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.620183 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:51.683633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.683242 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd7fe58-f59f-43f3-a31a-b45bab58a160-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd7fe58-f59f-43f3-a31a-b45bab58a160-plugins-config-volume\") pod \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\" (UID: \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\") " Nov 02 14:05:51.683633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.683316 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e904d1f-b4b5-42b1-82c2-5c0272d73582-kube-api-access-w2bbh\" (UniqueName: \"kubernetes.io/projected/2e904d1f-b4b5-42b1-82c2-5c0272d73582-kube-api-access-w2bbh\") pod \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\" (UID: \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\") " Nov 02 14:05:51.683633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.683441 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e904d1f-b4b5-42b1-82c2-5c0272d73582-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e904d1f-b4b5-42b1-82c2-5c0272d73582-plugins-config-volume\") pod \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\" (UID: \"2e904d1f-b4b5-42b1-82c2-5c0272d73582\") " Nov 02 14:05:51.683633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.683509 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd7fe58-f59f-43f3-a31a-b45bab58a160-kube-api-access-p8sms\" (UniqueName: \"kubernetes.io/projected/0bd7fe58-f59f-43f3-a31a-b45bab58a160-kube-api-access-p8sms\") pod \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\" (UID: \"0bd7fe58-f59f-43f3-a31a-b45bab58a160\") " Nov 02 14:05:51.685406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.685370 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e904d1f-b4b5-42b1-82c2-5c0272d73582-plugins-config-volume" pod "2e904d1f-b4b5-42b1-82c2-5c0272d73582" (UID: "2e904d1f-b4b5-42b1-82c2-5c0272d73582"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:51.685564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.685547 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd7fe58-f59f-43f3-a31a-b45bab58a160-plugins-config-volume" pod "0bd7fe58-f59f-43f3-a31a-b45bab58a160" (UID: "0bd7fe58-f59f-43f3-a31a-b45bab58a160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:05:51.689024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.688656 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd7fe58-f59f-43f3-a31a-b45bab58a160-kube-api-access-p8sms" pod "0bd7fe58-f59f-43f3-a31a-b45bab58a160" (UID: "0bd7fe58-f59f-43f3-a31a-b45bab58a160"). InnerVolumeSpecName "kube-api-access-p8sms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:51.691660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.690101 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e904d1f-b4b5-42b1-82c2-5c0272d73582-kube-api-access-w2bbh" pod "2e904d1f-b4b5-42b1-82c2-5c0272d73582" (UID: "2e904d1f-b4b5-42b1-82c2-5c0272d73582"). InnerVolumeSpecName "kube-api-access-w2bbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:51.749179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.749089 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:51.836232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.800360 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd7fe58-f59f-43f3-a31a-b45bab58a160-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:51.836232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.800414 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2bbh\" (UniqueName: \"kubernetes.io/projected/2e904d1f-b4b5-42b1-82c2-5c0272d73582-kube-api-access-w2bbh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:51.836232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.800433 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e904d1f-b4b5-42b1-82c2-5c0272d73582-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:51.836232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:51.800448 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8sms\" (UniqueName: \"kubernetes.io/projected/0bd7fe58-f59f-43f3-a31a-b45bab58a160-kube-api-access-p8sms\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:05:52.080474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.080417 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:52.251130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.251094 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:52.563807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.563770 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" Nov 02 14:05:52.565112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.565070 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6" event={"ID":"0bd7fe58-f59f-43f3-a31a-b45bab58a160","Type":"ContainerDied","Data":"454695292a6aeb75ac391678d0171f2723616041412c0b13cac1558e5c4c10d4"} Nov 02 14:05:52.565285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.565155 5591 scope.go:122] "RemoveContainer" containerID="31793f3d99aaf58235dc3d5bf573ea5502507863a678092df178ac52ba902cc0" Nov 02 14:05:52.566675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.566647 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" event={"ID":"2e904d1f-b4b5-42b1-82c2-5c0272d73582","Type":"ContainerDied","Data":"6453d0970e890516c32c254bb4e78eb98683923d81b6536a66d333a73f756545"} Nov 02 14:05:52.566742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.566662 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb" Nov 02 14:05:52.577283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.577264 5591 scope.go:122] "RemoveContainer" containerID="c9a0bafddbf4114738bea69c14066c47cc2cab3fa16b553be849804a9400cf52" Nov 02 14:05:52.620971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.620936 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb"] Nov 02 14:05:52.633859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.633835 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-856ccc4879-tdkfb"] Nov 02 14:05:52.662472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.662443 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6"] Nov 02 14:05:52.667161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:52.667137 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-5b8b686c47-jjwx6"] Nov 02 14:05:53.511004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.510389 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd7fe58-f59f-43f3-a31a-b45bab58a160" path="/var/lib/kubelet/pods/0bd7fe58-f59f-43f3-a31a-b45bab58a160/volumes" Nov 02 14:05:53.529007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.528682 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e904d1f-b4b5-42b1-82c2-5c0272d73582" path="/var/lib/kubelet/pods/2e904d1f-b4b5-42b1-82c2-5c0272d73582/volumes" Nov 02 14:05:53.602042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.598347 5591 generic.go:356] "Generic (PLEG): container finished" podID="7833de90-13be-4b90-a950-1d3083a72f92" containerID="979b5222ffb680c54a7d289f7bf416038308347e89358165da85caecac355d7b" exitCode=0 Nov 02 14:05:53.602042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.598433 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" event={"ID":"7833de90-13be-4b90-a950-1d3083a72f92","Type":"ContainerDied","Data":"979b5222ffb680c54a7d289f7bf416038308347e89358165da85caecac355d7b"} Nov 02 14:05:53.602042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.602022 5591 scope.go:122] "RemoveContainer" containerID="979b5222ffb680c54a7d289f7bf416038308347e89358165da85caecac355d7b" Nov 02 14:05:53.606426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.606386 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:53.606575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.606489 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:53.606575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.606541 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:05:53.875050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:53.874943 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:54.609974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:54.609941 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" event={"ID":"7833de90-13be-4b90-a950-1d3083a72f92","Type":"ContainerStarted","Data":"a774cc9237a4843827c2e00371cad31b8457bad82ad267d19155a91495fe9387"} Nov 02 14:05:55.466977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:55.466934 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:05:55.467363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:55.467334 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 14:05:55.467602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:55.467559 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 14:05:55.936442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:55.936407 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:56.130034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:56.129972 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:56.301099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:56.301014 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:56.467007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:56.466374 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 14:05:56.467007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:56.466438 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 14:05:57.473009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:57.471356 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 14:05:57.473009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:57.471415 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 14:05:57.911321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:57.911287 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:58.078004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.077838 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr"] Nov 02 14:05:58.085296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.078838 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2e904d1f-b4b5-42b1-82c2-5c0272d73582" containerName="payload-processing" Nov 02 14:05:58.085296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.078865 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e904d1f-b4b5-42b1-82c2-5c0272d73582" containerName="payload-processing" Nov 02 14:05:58.085296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.078902 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0bd7fe58-f59f-43f3-a31a-b45bab58a160" containerName="payload-pre-processing" Nov 02 14:05:58.085296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.078911 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd7fe58-f59f-43f3-a31a-b45bab58a160" containerName="payload-pre-processing" Nov 02 14:05:58.085296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.079134 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd7fe58-f59f-43f3-a31a-b45bab58a160" containerName="payload-pre-processing" Nov 02 14:05:58.085296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.079166 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e904d1f-b4b5-42b1-82c2-5c0272d73582" containerName="payload-processing" Nov 02 14:05:58.088038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.087328 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:58.136120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.109621 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr\" (UID: \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:58.136120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.110160 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcjg\" (UniqueName: \"kubernetes.io/projected/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-kube-api-access-nqcjg\") pod \"payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr\" (UID: \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:58.136120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.114002 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:58.136120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.135715 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-794667-dockercfg-wcgjj\"" Nov 02 14:05:58.136120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.135834 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-794667\"" Nov 02 14:05:58.136120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.135872 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr"] Nov 02 14:05:58.160840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.157419 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c"] Nov 02 14:05:58.176454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.176370 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:58.234014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.215849 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcjg\" (UniqueName: \"kubernetes.io/projected/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-kube-api-access-nqcjg\") pod \"payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr\" (UID: \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:58.234014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.216160 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr\" (UID: \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:58.236343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.236306 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr\" (UID: \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:58.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.338190 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e069b-84ac-4c68-9fa4-123d0a6a6417-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c\" (UID: \"003e069b-84ac-4c68-9fa4-123d0a6a6417\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:58.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.338386 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jnb\" (UniqueName: \"kubernetes.io/projected/003e069b-84ac-4c68-9fa4-123d0a6a6417-kube-api-access-b6jnb\") pod \"payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c\" (UID: \"003e069b-84ac-4c68-9fa4-123d0a6a6417\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:58.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.369230 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcjg\" (UniqueName: \"kubernetes.io/projected/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-kube-api-access-nqcjg\") pod \"payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr\" (UID: \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:58.475061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.467788 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jnb\" (UniqueName: \"kubernetes.io/projected/003e069b-84ac-4c68-9fa4-123d0a6a6417-kube-api-access-b6jnb\") pod \"payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c\" (UID: \"003e069b-84ac-4c68-9fa4-123d0a6a6417\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:58.497458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.497416 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e069b-84ac-4c68-9fa4-123d0a6a6417-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c\" (UID: \"003e069b-84ac-4c68-9fa4-123d0a6a6417\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:58.499067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.499041 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e069b-84ac-4c68-9fa4-123d0a6a6417-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c\" (UID: \"003e069b-84ac-4c68-9fa4-123d0a6a6417\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:58.512007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.505389 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 14:05:58.512007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.505455 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 14:05:58.613094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.612637 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c"] Nov 02 14:05:58.622099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.617662 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jnb\" (UniqueName: \"kubernetes.io/projected/003e069b-84ac-4c68-9fa4-123d0a6a6417-kube-api-access-b6jnb\") pod \"payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c\" (UID: \"003e069b-84ac-4c68-9fa4-123d0a6a6417\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:58.622099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.617907 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:58.822782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:58.822737 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:59.101014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.100151 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr"] Nov 02 14:05:59.284335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.284234 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c"] Nov 02 14:05:59.294903 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:05:59.294089 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003e069b_84ac_4c68_9fa4_123d0a6a6417.slice/crio-c3fdb1baea2d0c5495359357bc9fe11788616bedb86a4c9d3ab38a32c7463ec7 WatchSource:0}: Error finding container c3fdb1baea2d0c5495359357bc9fe11788616bedb86a4c9d3ab38a32c7463ec7: Status 404 returned error can't find the container with id c3fdb1baea2d0c5495359357bc9fe11788616bedb86a4c9d3ab38a32c7463ec7 Nov 02 14:05:59.468009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.466426 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 14:05:59.468009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.466489 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 14:05:59.566071 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.556126 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:05:59.688149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.688099 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" event={"ID":"e59169b9-f374-4c2c-9396-95c7b4cb3a9f","Type":"ContainerStarted","Data":"c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6"} Nov 02 14:05:59.688323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.688210 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" event={"ID":"e59169b9-f374-4c2c-9396-95c7b4cb3a9f","Type":"ContainerStarted","Data":"ad967985af79401b26aa67fb343afacdd92581a6ac01423f257cd728e74da9d5"} Nov 02 14:05:59.688323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.688275 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:05:59.689697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.689674 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" event={"ID":"003e069b-84ac-4c68-9fa4-123d0a6a6417","Type":"ContainerStarted","Data":"57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6"} Nov 02 14:05:59.689852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.689840 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" event={"ID":"003e069b-84ac-4c68-9fa4-123d0a6a6417","Type":"ContainerStarted","Data":"c3fdb1baea2d0c5495359357bc9fe11788616bedb86a4c9d3ab38a32c7463ec7"} Nov 02 14:05:59.689929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.689921 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:05:59.771010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.762259 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" podStartSLOduration=1.7622389269999998 podStartE2EDuration="1.762238927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:59.75219253 +0000 UTC m=+1308.477814300" watchObservedRunningTime="2026-11-02 14:05:59.762238927 +0000 UTC m=+1308.487860701" Nov 02 14:05:59.803117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:05:59.803067 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" podStartSLOduration=1.803053867 podStartE2EDuration="1.803053867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:59.799679763 +0000 UTC m=+1308.525301532" watchObservedRunningTime="2026-11-02 14:05:59.803053867 +0000 UTC m=+1308.528675636" Nov 02 14:06:00.413064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:00.411024 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:00.457047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:00.456228 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:06:00.462076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:00.457655 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:06:00.683629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:00.683519 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:06:00.686493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:00.686464 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:06:00.689363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:00.689337 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:06:01.074561 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:01.074520 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:06:02.165046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:02.157260 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:04.056814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:04.053226 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:05.304681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.304632 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:05.488034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.477517 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:05.604078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.579505 5591 patch_prober.go:29] interesting pod/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 14:06:05.604078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.579565 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 14:06:05.766270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.766091 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82"] Nov 02 14:06:05.789372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.789333 5591 generic.go:356] "Generic (PLEG): container finished" podID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerID="39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2" exitCode=0 Nov 02 14:06:05.935238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.935203 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82"] Nov 02 14:06:05.935523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.935495 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" event={"ID":"33cb9750-f264-4b5f-8069-8af98fdc5001","Type":"ContainerDied","Data":"39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2"} Nov 02 14:06:05.941965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.941914 5591 scope.go:122] "RemoveContainer" containerID="39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2" Nov 02 14:06:05.944390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:05.944369 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:06.131032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.128069 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-17a7ba-dockercfg-d998c\"" Nov 02 14:06:06.161055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.145860 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-17a7ba\"" Nov 02 14:06:06.175005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.171701 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:06:06.175005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.172483 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:06:06.175005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.173035 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:06:06.180864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.175613 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfm8m\" (UniqueName: \"kubernetes.io/projected/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-kube-api-access-xfm8m\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82\" (UID: \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:06.180864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.175748 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82\" (UID: \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:06.281532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.281502 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfm8m\" (UniqueName: \"kubernetes.io/projected/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-kube-api-access-xfm8m\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82\" (UID: \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:06.281655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.281585 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82\" (UID: \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:06.282242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.282223 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82\" (UID: \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:06.368008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.337834 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfm8m\" (UniqueName: \"kubernetes.io/projected/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-kube-api-access-xfm8m\") pod \"payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82\" (UID: \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:06.427502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.427459 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:06:06.482403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.482290 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:06.491526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.491241 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:06.806033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:06.802769 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" event={"ID":"33cb9750-f264-4b5f-8069-8af98fdc5001","Type":"ContainerStarted","Data":"537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2"} Nov 02 14:06:07.028380 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:07.028350 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82"] Nov 02 14:06:07.029078 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:06:07.029053 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa0bbfa_7b4f_46b6_8850_7e425d9d2a66.slice/crio-d73c7aa96db3c0b8a9ce533a7b6552a1d94895fabaf6ebdaf8cd609ad90ec62d WatchSource:0}: Error finding container d73c7aa96db3c0b8a9ce533a7b6552a1d94895fabaf6ebdaf8cd609ad90ec62d: Status 404 returned error can't find the container with id d73c7aa96db3c0b8a9ce533a7b6552a1d94895fabaf6ebdaf8cd609ad90ec62d Nov 02 14:06:07.206575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:07.206541 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:06:07.207485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:07.207452 5591 patch_prober.go:29] interesting pod/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 14:06:07.207606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:07.207513 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 14:06:07.669789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:07.669738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:07.811845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:07.811796 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" event={"ID":"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66","Type":"ContainerStarted","Data":"448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829"} Nov 02 14:06:07.812098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:07.811865 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" event={"ID":"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66","Type":"ContainerStarted","Data":"d73c7aa96db3c0b8a9ce533a7b6552a1d94895fabaf6ebdaf8cd609ad90ec62d"} Nov 02 14:06:07.812098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:07.812051 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:08.206551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:08.206514 5591 patch_prober.go:29] interesting pod/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 14:06:08.206739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:08.206568 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 14:06:08.397254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:08.397216 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:08.849575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:08.849532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:09.052167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:09.052038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:09.206411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:09.206375 5591 patch_prober.go:29] interesting pod/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 14:06:09.206689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:09.206660 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 14:06:09.218798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:09.218748 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:10.207033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:10.206963 5591 patch_prober.go:29] interesting pod/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 14:06:10.207448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:10.207054 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 14:06:10.702295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:10.702261 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:06:10.702464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:10.702333 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:06:10.759540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:10.759444 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" podStartSLOduration=5.759424993 podStartE2EDuration="5.759424993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:07.862351416 +0000 UTC m=+1316.587973199" watchObservedRunningTime="2026-11-02 14:06:10.759424993 +0000 UTC m=+1319.485046789" Nov 02 14:06:10.759870 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:10.759841 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:11.206742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:11.206705 5591 patch_prober.go:29] interesting pod/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 14:06:11.206964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:11.206768 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 14:06:12.206465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:12.206421 5591 patch_prober.go:29] interesting pod/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 02 14:06:12.206943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:12.206483 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 02 14:06:12.595559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:12.592269 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:13.205855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:13.205797 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:13.210163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:13.210138 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:06:13.210567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:13.210548 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:06:13.211285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:13.211266 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:06:13.386264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:13.386227 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:14.126696 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:14.126655 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf"] Nov 02 14:06:14.126967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:14.126897 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" containerID="cri-o://95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c" gracePeriod=30 Nov 02 14:06:14.356966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:14.356926 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:14.955308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:14.955272 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:15.226052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:15.225111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:16.136391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:16.136363 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:17.139598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:17.139560 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n"] Nov 02 14:06:17.184509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:17.184473 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2"] Nov 02 14:06:17.343211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:17.343178 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:17.512701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:17.511704 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:18.009839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:18.009786 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:18.822023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:18.821978 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:06:19.398520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.398460 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 14:06:19.498842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.498797 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9"] Nov 02 14:06:19.499545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.499521 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" Nov 02 14:06:19.499545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.499540 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" Nov 02 14:06:19.499746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.499650 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" containerName="istio-proxy" Nov 02 14:06:19.503222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.503201 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.521898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.521875 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-794667-serving-cert\"" Nov 02 14:06:19.531863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.531837 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9"] Nov 02 14:06:19.534076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.534037 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-socket\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.534268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.534249 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-credential-socket\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.534397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.534382 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b357daf9-2941-45c8-bd2f-ac9324282013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b357daf9-2941-45c8-bd2f-ac9324282013-istio-podinfo\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.534516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.534502 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-istio-token\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-istio-token\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.534692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.534678 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-envoy\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.534805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.534792 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-kube-api-access-xsxbr\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-kube-api-access-xsxbr\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.535043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.534972 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b357daf9-2941-45c8-bd2f-ac9324282013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b357daf9-2941-45c8-bd2f-ac9324282013-istiod-ca-cert\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.535212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.535195 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-certs\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.535504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.535489 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-data\") pod \"b357daf9-2941-45c8-bd2f-ac9324282013\" (UID: \"b357daf9-2941-45c8-bd2f-ac9324282013\") " Nov 02 14:06:19.537702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.537680 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-socket" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:06:19.538178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.538152 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b357daf9-2941-45c8-bd2f-ac9324282013-istiod-ca-cert" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:06:19.538413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.538391 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-certs" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:06:19.538592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.538574 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-data" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:06:19.539374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.539341 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-credential-socket" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:06:19.550830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.549352 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b357daf9-2941-45c8-bd2f-ac9324282013-istio-podinfo" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:06:19.557732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.557706 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-istio-token" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:19.564556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.564521 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-kube-api-access-xsxbr" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "kube-api-access-xsxbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:19.576182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.576138 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-envoy" pod "b357daf9-2941-45c8-bd2f-ac9324282013" (UID: "b357daf9-2941-45c8-bd2f-ac9324282013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:06:19.638230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638197 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-metrics-certs\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.638396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638351 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.638550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638533 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwqm\" (UniqueName: \"kubernetes.io/projected/6e58081c-1e1b-4aee-b25d-f012110fc2dd-kube-api-access-wjwqm\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.638656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638641 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e58081c-1e1b-4aee-b25d-f012110fc2dd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.638714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638705 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.638757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638717 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxbr\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-kube-api-access-xsxbr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.638757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638727 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b357daf9-2941-45c8-bd2f-ac9324282013-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.638757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638736 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.638889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638759 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.638889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638781 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.638889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638794 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b357daf9-2941-45c8-bd2f-ac9324282013-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.638889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638820 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b357daf9-2941-45c8-bd2f-ac9324282013-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.638889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.638829 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b357daf9-2941-45c8-bd2f-ac9324282013-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:19.643633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.643609 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:19.740053 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.739943 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwqm\" (UniqueName: \"kubernetes.io/projected/6e58081c-1e1b-4aee-b25d-f012110fc2dd-kube-api-access-wjwqm\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.757290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.757252 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e58081c-1e1b-4aee-b25d-f012110fc2dd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.758493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.758464 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-metrics-certs\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.759296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.759273 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.759810 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:19.759521 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-serving-cert: secret "maas-api-e2e-ipp-a-794667-serving-cert" not found Nov 02 14:06:19.760014 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:19.759996 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls podName:6e58081c-1e1b-4aee-b25d-f012110fc2dd nodeName:}" failed. No retries permitted until 2026-11-02 14:06:20.259954847 +0000 UTC m=+1328.985576610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls") pod "maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" (UID: "6e58081c-1e1b-4aee-b25d-f012110fc2dd") : secret "maas-api-e2e-ipp-a-794667-serving-cert" not found Nov 02 14:06:19.760144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.758362 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e58081c-1e1b-4aee-b25d-f012110fc2dd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.788741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.788704 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-metrics-certs\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.792314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.792278 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwqm\" (UniqueName: \"kubernetes.io/projected/6e58081c-1e1b-4aee-b25d-f012110fc2dd-kube-api-access-wjwqm\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:19.826200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:19.826160 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:20.001138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.001058 5591 generic.go:356] "Generic (PLEG): container finished" podID="b357daf9-2941-45c8-bd2f-ac9324282013" containerID="95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c" exitCode=0 Nov 02 14:06:20.001138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.001128 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" Nov 02 14:06:20.001321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.001143 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" event={"ID":"b357daf9-2941-45c8-bd2f-ac9324282013","Type":"ContainerDied","Data":"95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c"} Nov 02 14:06:20.001321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.001211 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf" event={"ID":"b357daf9-2941-45c8-bd2f-ac9324282013","Type":"ContainerDied","Data":"e483954d24979eb2c2ec09c2c12f8d1a91c03324addda20a4ea7a6e9f8f2debb"} Nov 02 14:06:20.001321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.001242 5591 scope.go:122] "RemoveContainer" containerID="95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c" Nov 02 14:06:20.011615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.011596 5591 scope.go:122] "RemoveContainer" containerID="95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c" Nov 02 14:06:20.011909 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:20.011889 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c\": container with ID starting with 95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c not found: ID does not exist" containerID="95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c" Nov 02 14:06:20.011993 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.011920 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c"} err="failed to get container status \"95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c\": rpc error: code = NotFound desc = could not find container \"95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c\": container with ID starting with 95c0d27a3b5583a88598d21890832b76586b722b90a7c88b922e668a14e9237c not found: ID does not exist" Nov 02 14:06:20.013698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.013676 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:20.060275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.060240 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf"] Nov 02 14:06:20.064221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.064195 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-fpfvf"] Nov 02 14:06:20.267278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.267188 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:20.269655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.269625 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:20.445300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.443931 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:20.802064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:20.799863 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9"] Nov 02 14:06:20.859282 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:06:20.857094 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e58081c_1e1b_4aee_b25d_f012110fc2dd.slice/crio-d936212ce23217eb2db28a8e4837641ec0403aee424bcd0ab3f665c5fa041877 WatchSource:0}: Error finding container d936212ce23217eb2db28a8e4837641ec0403aee424bcd0ab3f665c5fa041877: Status 404 returned error can't find the container with id d936212ce23217eb2db28a8e4837641ec0403aee424bcd0ab3f665c5fa041877 Nov 02 14:06:21.009878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:21.009840 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" event={"ID":"6e58081c-1e1b-4aee-b25d-f012110fc2dd","Type":"ContainerStarted","Data":"d936212ce23217eb2db28a8e4837641ec0403aee424bcd0ab3f665c5fa041877"} Nov 02 14:06:21.432688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:21.432654 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b357daf9-2941-45c8-bd2f-ac9324282013" path="/var/lib/kubelet/pods/b357daf9-2941-45c8-bd2f-ac9324282013/volumes" Nov 02 14:06:21.470118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:21.460448 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:22.031014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.027170 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" event={"ID":"6e58081c-1e1b-4aee-b25d-f012110fc2dd","Type":"ContainerStarted","Data":"94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57"} Nov 02 14:06:22.031014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.027395 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:22.080595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.080528 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" podStartSLOduration=2.750472508 podStartE2EDuration="3.080508684s" totalImagesPullingTime="330.036176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:22.079796042 +0000 UTC m=+1330.805417811" watchObservedRunningTime="2026-11-02 14:06:22.080508684 +0000 UTC m=+1330.806130455" Nov 02 14:06:22.140137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.140097 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" podUID="0dd01d69-da94-420f-b1ec-ce6236df44c4" containerName="payload-processing" containerID="cri-o://216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c" gracePeriod=25 Nov 02 14:06:22.184731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.184694 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" podUID="c7ba4b1c-559d-49de-9796-c489426d2a07" containerName="payload-pre-processing" containerID="cri-o://188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122" gracePeriod=25 Nov 02 14:06:22.187334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.187302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:22.354852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.354810 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:22.488011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.487850 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:06:22.528106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.528078 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd01d69-da94-420f-b1ec-ce6236df44c4-kube-api-access-dcbpp\" (UniqueName: \"kubernetes.io/projected/0dd01d69-da94-420f-b1ec-ce6236df44c4-kube-api-access-dcbpp\") pod \"0dd01d69-da94-420f-b1ec-ce6236df44c4\" (UID: \"0dd01d69-da94-420f-b1ec-ce6236df44c4\") " Nov 02 14:06:22.528227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.528197 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd01d69-da94-420f-b1ec-ce6236df44c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd01d69-da94-420f-b1ec-ce6236df44c4-plugins-config-volume\") pod \"0dd01d69-da94-420f-b1ec-ce6236df44c4\" (UID: \"0dd01d69-da94-420f-b1ec-ce6236df44c4\") " Nov 02 14:06:22.528605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.528563 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd01d69-da94-420f-b1ec-ce6236df44c4-plugins-config-volume" pod "0dd01d69-da94-420f-b1ec-ce6236df44c4" (UID: "0dd01d69-da94-420f-b1ec-ce6236df44c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:06:22.529056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.529029 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd01d69-da94-420f-b1ec-ce6236df44c4-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:22.530733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.530713 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd01d69-da94-420f-b1ec-ce6236df44c4-kube-api-access-dcbpp" pod "0dd01d69-da94-420f-b1ec-ce6236df44c4" (UID: "0dd01d69-da94-420f-b1ec-ce6236df44c4"). InnerVolumeSpecName "kube-api-access-dcbpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:22.550702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.550639 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:06:22.629819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.629787 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ba4b1c-559d-49de-9796-c489426d2a07-kube-api-access-6cpz7\" (UniqueName: \"kubernetes.io/projected/c7ba4b1c-559d-49de-9796-c489426d2a07-kube-api-access-6cpz7\") pod \"c7ba4b1c-559d-49de-9796-c489426d2a07\" (UID: \"c7ba4b1c-559d-49de-9796-c489426d2a07\") " Nov 02 14:06:22.630074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.630052 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ba4b1c-559d-49de-9796-c489426d2a07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7ba4b1c-559d-49de-9796-c489426d2a07-plugins-config-volume\") pod \"c7ba4b1c-559d-49de-9796-c489426d2a07\" (UID: \"c7ba4b1c-559d-49de-9796-c489426d2a07\") " Nov 02 14:06:22.630464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.630428 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ba4b1c-559d-49de-9796-c489426d2a07-plugins-config-volume" pod "c7ba4b1c-559d-49de-9796-c489426d2a07" (UID: "c7ba4b1c-559d-49de-9796-c489426d2a07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:06:22.630817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.630794 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbpp\" (UniqueName: \"kubernetes.io/projected/0dd01d69-da94-420f-b1ec-ce6236df44c4-kube-api-access-dcbpp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:22.630910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.630822 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7ba4b1c-559d-49de-9796-c489426d2a07-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:22.632214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.632186 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ba4b1c-559d-49de-9796-c489426d2a07-kube-api-access-6cpz7" pod "c7ba4b1c-559d-49de-9796-c489426d2a07" (UID: "c7ba4b1c-559d-49de-9796-c489426d2a07"). InnerVolumeSpecName "kube-api-access-6cpz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:22.732480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:22.732444 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpz7\" (UniqueName: \"kubernetes.io/projected/c7ba4b1c-559d-49de-9796-c489426d2a07-kube-api-access-6cpz7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:23.077052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.075288 5591 generic.go:356] "Generic (PLEG): container finished" podID="c7ba4b1c-559d-49de-9796-c489426d2a07" containerID="188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122" exitCode=0 Nov 02 14:06:23.077052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.075432 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" event={"ID":"c7ba4b1c-559d-49de-9796-c489426d2a07","Type":"ContainerDied","Data":"188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122"} Nov 02 14:06:23.077052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.075493 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" event={"ID":"c7ba4b1c-559d-49de-9796-c489426d2a07","Type":"ContainerDied","Data":"7061bba0f65ad73c57de683f689c2f8d7ccfc79fb62d58c01390573e2680b3b2"} Nov 02 14:06:23.077052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.075523 5591 scope.go:122] "RemoveContainer" containerID="188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122" Nov 02 14:06:23.077052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.075697 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2" Nov 02 14:06:23.095616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.095550 5591 generic.go:356] "Generic (PLEG): container finished" podID="0dd01d69-da94-420f-b1ec-ce6236df44c4" containerID="216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c" exitCode=0 Nov 02 14:06:23.097013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.096968 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" Nov 02 14:06:23.104146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.098224 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" event={"ID":"0dd01d69-da94-420f-b1ec-ce6236df44c4","Type":"ContainerDied","Data":"216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c"} Nov 02 14:06:23.104146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.098275 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n" event={"ID":"0dd01d69-da94-420f-b1ec-ce6236df44c4","Type":"ContainerDied","Data":"108f7209d132294b3f6f5be2f3d77f28ff6f5c0aec133972bfe32561a28c84cb"} Nov 02 14:06:23.132782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.132134 5591 scope.go:122] "RemoveContainer" containerID="188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122" Nov 02 14:06:23.134846 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:23.134525 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122\": container with ID starting with 188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122 not found: ID does not exist" containerID="188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122" Nov 02 14:06:23.134846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.134566 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122"} err="failed to get container status \"188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122\": rpc error: code = NotFound desc = could not find container \"188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122\": container with ID starting with 188e9fb3d2439a0088bac297872d373bd743a91e98dd8c3593fd0a31936e3122 not found: ID does not exist" Nov 02 14:06:23.134846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.134585 5591 scope.go:122] "RemoveContainer" containerID="216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c" Nov 02 14:06:23.146614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.146427 5591 scope.go:122] "RemoveContainer" containerID="216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c" Nov 02 14:06:23.146780 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:23.146756 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c\": container with ID starting with 216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c not found: ID does not exist" containerID="216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c" Nov 02 14:06:23.146853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.146792 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c"} err="failed to get container status \"216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c\": rpc error: code = NotFound desc = could not find container \"216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c\": container with ID starting with 216e78d0ddb35ce3ec9bfe6277f106ad3d02e5e9f94aaa8a122e978196e36b7c not found: ID does not exist" Nov 02 14:06:23.165015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.164972 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n"] Nov 02 14:06:23.177598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.177573 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-77dc6d9bb7-sx69n"] Nov 02 14:06:23.228510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.226050 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:23.242448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.235139 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2"] Nov 02 14:06:23.242448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.242153 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-86c55bc995-6kvh2"] Nov 02 14:06:23.414358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.414323 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:23.433854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.433824 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd01d69-da94-420f-b1ec-ce6236df44c4" path="/var/lib/kubelet/pods/0dd01d69-da94-420f-b1ec-ce6236df44c4/volumes" Nov 02 14:06:23.434351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:23.434334 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ba4b1c-559d-49de-9796-c489426d2a07" path="/var/lib/kubelet/pods/c7ba4b1c-559d-49de-9796-c489426d2a07/volumes" Nov 02 14:06:24.041443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:24.041405 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:24.457002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:24.456214 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:25.632863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:25.631759 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:27.399932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:27.399896 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:27.801750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:27.801667 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:27.997009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:27.996950 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:28.103435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:28.103356 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:06:29.481263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.481228 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:29.805060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.733409 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:29.920319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.920286 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm"] Nov 02 14:06:29.966361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.966317 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0dd01d69-da94-420f-b1ec-ce6236df44c4" containerName="payload-processing" Nov 02 14:06:29.966623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.966597 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd01d69-da94-420f-b1ec-ce6236df44c4" containerName="payload-processing" Nov 02 14:06:29.980082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.980049 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7ba4b1c-559d-49de-9796-c489426d2a07" containerName="payload-pre-processing" Nov 02 14:06:29.980368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.980344 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ba4b1c-559d-49de-9796-c489426d2a07" containerName="payload-pre-processing" Nov 02 14:06:29.997181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.997146 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd01d69-da94-420f-b1ec-ce6236df44c4" containerName="payload-processing" Nov 02 14:06:29.997420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:29.997408 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ba4b1c-559d-49de-9796-c489426d2a07" containerName="payload-pre-processing" Nov 02 14:06:30.003866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.003842 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm"] Nov 02 14:06:30.004031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.003937 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:30.035237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.035204 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b7489f\"" Nov 02 14:06:30.035237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.035225 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b7489f-dockercfg-xxspv\"" Nov 02 14:06:30.143360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.143320 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppsz\" (UniqueName: \"kubernetes.io/projected/c75085e6-f3ae-403e-a714-095deb1ea874-kube-api-access-wppsz\") pod \"payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm\" (UID: \"c75085e6-f3ae-403e-a714-095deb1ea874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:30.143554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.143535 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75085e6-f3ae-403e-a714-095deb1ea874-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm\" (UID: \"c75085e6-f3ae-403e-a714-095deb1ea874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:30.244816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.244777 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppsz\" (UniqueName: \"kubernetes.io/projected/c75085e6-f3ae-403e-a714-095deb1ea874-kube-api-access-wppsz\") pod \"payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm\" (UID: \"c75085e6-f3ae-403e-a714-095deb1ea874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:30.245147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.245052 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75085e6-f3ae-403e-a714-095deb1ea874-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm\" (UID: \"c75085e6-f3ae-403e-a714-095deb1ea874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:30.245714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.245691 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75085e6-f3ae-403e-a714-095deb1ea874-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm\" (UID: \"c75085e6-f3ae-403e-a714-095deb1ea874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:30.270343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.270305 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppsz\" (UniqueName: \"kubernetes.io/projected/c75085e6-f3ae-403e-a714-095deb1ea874-kube-api-access-wppsz\") pod \"payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm\" (UID: \"c75085e6-f3ae-403e-a714-095deb1ea874\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:30.318677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.318639 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:30.628820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:30.628794 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm"] Nov 02 14:06:30.630760 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:06:30.630725 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75085e6_f3ae_403e_a714_095deb1ea874.slice/crio-000ba9e9c9c56955b1601259e54ff7c9e010080bdc2d523c7091d2477cbb439a WatchSource:0}: Error finding container 000ba9e9c9c56955b1601259e54ff7c9e010080bdc2d523c7091d2477cbb439a: Status 404 returned error can't find the container with id 000ba9e9c9c56955b1601259e54ff7c9e010080bdc2d523c7091d2477cbb439a Nov 02 14:06:31.074856 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:31.074821 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:06:31.176159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:31.176126 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" event={"ID":"c75085e6-f3ae-403e-a714-095deb1ea874","Type":"ContainerStarted","Data":"7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292"} Nov 02 14:06:31.176307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:31.176177 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" event={"ID":"c75085e6-f3ae-403e-a714-095deb1ea874","Type":"ContainerStarted","Data":"000ba9e9c9c56955b1601259e54ff7c9e010080bdc2d523c7091d2477cbb439a"} Nov 02 14:06:31.176307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:31.176276 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:31.221038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:31.220965 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" podStartSLOduration=2.220949976 podStartE2EDuration="2.220949976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:31.220043476 +0000 UTC m=+1339.945665246" watchObservedRunningTime="2026-11-02 14:06:31.220949976 +0000 UTC m=+1339.946571746" Nov 02 14:06:32.009010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:31.992735 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:34.290013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:34.282605 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:36.307009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:36.306659 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:38.168724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:38.168688 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:38.340014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:38.337636 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:40.097055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:40.097011 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:41.615227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:41.615191 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8"] Nov 02 14:06:41.637234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:41.637202 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6"] Nov 02 14:06:41.811842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:41.811803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:42.084038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:42.083966 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:42.195769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:42.194909 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:06:42.272559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:42.272527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:43.369346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.369313 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:43.580677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.580418 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:43.776131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.776059 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww"] Nov 02 14:06:43.811312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.811274 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:43.830218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.812359 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww"] Nov 02 14:06:43.854271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.848834 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea44bf2f-bc55-4b19-8e44-42f3072271c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww\" (UID: \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:43.854271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.849049 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmh8\" (UniqueName: \"kubernetes.io/projected/ea44bf2f-bc55-4b19-8e44-42f3072271c8-kube-api-access-2fmh8\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww\" (UID: \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:43.854271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.854108 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fb206f\"" Nov 02 14:06:43.854691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.854343 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fb206f-dockercfg-hztpk\"" Nov 02 14:06:43.954020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.951340 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea44bf2f-bc55-4b19-8e44-42f3072271c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww\" (UID: \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:43.954020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.951524 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmh8\" (UniqueName: \"kubernetes.io/projected/ea44bf2f-bc55-4b19-8e44-42f3072271c8-kube-api-access-2fmh8\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww\" (UID: \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:43.954020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.953255 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea44bf2f-bc55-4b19-8e44-42f3072271c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww\" (UID: \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:43.980703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:43.980671 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmh8\" (UniqueName: \"kubernetes.io/projected/ea44bf2f-bc55-4b19-8e44-42f3072271c8-kube-api-access-2fmh8\") pod \"payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww\" (UID: \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:44.274221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:44.274186 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:44.556813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:44.556728 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:44.582997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:44.582957 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww"] Nov 02 14:06:44.583105 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:06:44.582966 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea44bf2f_bc55_4b19_8e44_42f3072271c8.slice/crio-a85a346ba488349c692c1b5ddb72dd1aad7d2e754feffc87ede29deca423c7cd WatchSource:0}: Error finding container a85a346ba488349c692c1b5ddb72dd1aad7d2e754feffc87ede29deca423c7cd: Status 404 returned error can't find the container with id a85a346ba488349c692c1b5ddb72dd1aad7d2e754feffc87ede29deca423c7cd Nov 02 14:06:45.324196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:45.324148 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" event={"ID":"ea44bf2f-bc55-4b19-8e44-42f3072271c8","Type":"ContainerStarted","Data":"3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02"} Nov 02 14:06:45.324471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:45.324205 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" event={"ID":"ea44bf2f-bc55-4b19-8e44-42f3072271c8","Type":"ContainerStarted","Data":"a85a346ba488349c692c1b5ddb72dd1aad7d2e754feffc87ede29deca423c7cd"} Nov 02 14:06:45.324471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:45.324241 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:45.367462 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:45.367412 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" podStartSLOduration=2.36739873 podStartE2EDuration="2.36739873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:45.366549031 +0000 UTC m=+1354.092170803" watchObservedRunningTime="2026-11-02 14:06:45.36739873 +0000 UTC m=+1354.093020522" Nov 02 14:06:45.853542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:45.853501 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:06:46.252555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:46.252516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:46.616099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:46.615991 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" podUID="7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b" containerName="payload-processing" containerID="cri-o://76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5" gracePeriod=25 Nov 02 14:06:46.637949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:46.637918 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" podUID="c7af67e4-3142-44e1-95cc-b69cbd3d46f1" containerName="payload-pre-processing" containerID="cri-o://fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d" gracePeriod=25 Nov 02 14:06:47.185034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.172699 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:06:47.219705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.219680 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:06:47.299420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.299391 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-kube-api-access-cplbc\" (UniqueName: \"kubernetes.io/projected/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-kube-api-access-cplbc\") pod \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\" (UID: \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\") " Nov 02 14:06:47.299749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.299577 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-kube-api-access-whmzd\") pod \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\" (UID: \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\") " Nov 02 14:06:47.299749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.299675 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-plugins-config-volume\") pod \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\" (UID: \"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b\") " Nov 02 14:06:47.299749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.299737 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-plugins-config-volume\") pod \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\" (UID: \"c7af67e4-3142-44e1-95cc-b69cbd3d46f1\") " Nov 02 14:06:47.300240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.300210 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-plugins-config-volume" pod "7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b" (UID: "7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:06:47.300465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.300368 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-plugins-config-volume" pod "c7af67e4-3142-44e1-95cc-b69cbd3d46f1" (UID: "c7af67e4-3142-44e1-95cc-b69cbd3d46f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:06:47.300529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.300494 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:47.300529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.300513 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:47.302333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.302261 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-kube-api-access-whmzd" pod "7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b" (UID: "7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b"). InnerVolumeSpecName "kube-api-access-whmzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:47.302929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.302899 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-kube-api-access-cplbc" pod "c7af67e4-3142-44e1-95cc-b69cbd3d46f1" (UID: "c7af67e4-3142-44e1-95cc-b69cbd3d46f1"). InnerVolumeSpecName "kube-api-access-cplbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:47.343281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.343247 5591 generic.go:356] "Generic (PLEG): container finished" podID="c7af67e4-3142-44e1-95cc-b69cbd3d46f1" containerID="fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d" exitCode=0 Nov 02 14:06:47.343419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.343311 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" Nov 02 14:06:47.343419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.343339 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" event={"ID":"c7af67e4-3142-44e1-95cc-b69cbd3d46f1","Type":"ContainerDied","Data":"fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d"} Nov 02 14:06:47.343419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.343394 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6" event={"ID":"c7af67e4-3142-44e1-95cc-b69cbd3d46f1","Type":"ContainerDied","Data":"a8ce82cf38fc0b869bc1dd50da5522284355f982258b0f1bd7d63d85d5eb62a7"} Nov 02 14:06:47.343616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.343422 5591 scope.go:122] "RemoveContainer" containerID="fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d" Nov 02 14:06:47.344972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.344946 5591 generic.go:356] "Generic (PLEG): container finished" podID="7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b" containerID="76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5" exitCode=0 Nov 02 14:06:47.345103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.345055 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" event={"ID":"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b","Type":"ContainerDied","Data":"76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5"} Nov 02 14:06:47.345103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.345069 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" Nov 02 14:06:47.345199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.345100 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8" event={"ID":"7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b","Type":"ContainerDied","Data":"c3a117e119d0ac9528e8354a3667361abb536b2b6ec396c24490d499fa785844"} Nov 02 14:06:47.366810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.366787 5591 scope.go:122] "RemoveContainer" containerID="fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d" Nov 02 14:06:47.367202 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:47.367182 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d\": container with ID starting with fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d not found: ID does not exist" containerID="fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d" Nov 02 14:06:47.367272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.367210 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d"} err="failed to get container status \"fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d\": rpc error: code = NotFound desc = could not find container \"fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d\": container with ID starting with fc612d5ce0c5050bcf57f65515dfea0df805faffa1ac23abc87eceae2593ef2d not found: ID does not exist" Nov 02 14:06:47.367272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.367227 5591 scope.go:122] "RemoveContainer" containerID="76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5" Nov 02 14:06:47.376541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.376522 5591 scope.go:122] "RemoveContainer" containerID="76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5" Nov 02 14:06:47.376806 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:47.376787 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5\": container with ID starting with 76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5 not found: ID does not exist" containerID="76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5" Nov 02 14:06:47.376853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.376814 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5"} err="failed to get container status \"76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5\": rpc error: code = NotFound desc = could not find container \"76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5\": container with ID starting with 76b267ac60c147a5f230dd211642dda6dfb88784ee2aadcd055c2454f4a252b5 not found: ID does not exist" Nov 02 14:06:47.401454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.401433 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b-kube-api-access-whmzd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:47.401454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.401455 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cplbc\" (UniqueName: \"kubernetes.io/projected/c7af67e4-3142-44e1-95cc-b69cbd3d46f1-kube-api-access-cplbc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:47.415608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.415543 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8"] Nov 02 14:06:47.433018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.432963 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-58f7b6697-h56g8"] Nov 02 14:06:47.457870 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.457797 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6"] Nov 02 14:06:47.462511 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:47.462488 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-697b7bf5f9-bfhc6"] Nov 02 14:06:48.086006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:48.085318 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:49.485851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:49.485813 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b" path="/var/lib/kubelet/pods/7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b/volumes" Nov 02 14:06:49.487156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:49.487131 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7af67e4-3142-44e1-95cc-b69cbd3d46f1" path="/var/lib/kubelet/pods/c7af67e4-3142-44e1-95cc-b69cbd3d46f1/volumes" Nov 02 14:06:49.685953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:49.685911 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:51.559007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:51.557422 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:51.891219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:51.891184 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c"] Nov 02 14:06:51.937978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:51.937944 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr"] Nov 02 14:06:52.083178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:52.083143 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:52.248323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:52.248226 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:52.414359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:52.414328 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:53.776937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:53.776839 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:55.435321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:55.435289 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:55.602008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:55.601167 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:56.335727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.335696 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:06:56.578551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.578506 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4"] Nov 02 14:06:56.579846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.579794 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" containerID="cri-o://a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036" gracePeriod=30 Nov 02 14:06:56.585497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.585468 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv"] Nov 02 14:06:56.585855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.585790 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" containerID="cri-o://e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7" gracePeriod=30 Nov 02 14:06:56.608448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.603372 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9"] Nov 02 14:06:56.609339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.609240 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" containerID="cri-o://46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15" gracePeriod=30 Nov 02 14:06:56.892055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.892006 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" podUID="003e069b-84ac-4c68-9fa4-123d0a6a6417" containerName="payload-processing" containerID="cri-o://57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6" gracePeriod=25 Nov 02 14:06:56.938495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:56.938457 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" podUID="e59169b9-f374-4c2c-9396-95c7b4cb3a9f" containerName="payload-pre-processing" containerID="cri-o://c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6" gracePeriod=25 Nov 02 14:06:57.218258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.215993 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:57.449522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.449494 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:06:57.454017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.453975 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:06:57.556971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.556861 5591 generic.go:356] "Generic (PLEG): container finished" podID="e59169b9-f374-4c2c-9396-95c7b4cb3a9f" containerID="c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6" exitCode=0 Nov 02 14:06:57.556971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.556919 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" Nov 02 14:06:57.556971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.556942 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" event={"ID":"e59169b9-f374-4c2c-9396-95c7b4cb3a9f","Type":"ContainerDied","Data":"c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6"} Nov 02 14:06:57.557325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.557036 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr" event={"ID":"e59169b9-f374-4c2c-9396-95c7b4cb3a9f","Type":"ContainerDied","Data":"ad967985af79401b26aa67fb343afacdd92581a6ac01423f257cd728e74da9d5"} Nov 02 14:06:57.557325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.557071 5591 scope.go:122] "RemoveContainer" containerID="c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6" Nov 02 14:06:57.559266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.559237 5591 generic.go:356] "Generic (PLEG): container finished" podID="003e069b-84ac-4c68-9fa4-123d0a6a6417" containerID="57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6" exitCode=0 Nov 02 14:06:57.559397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.559288 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" Nov 02 14:06:57.559397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.559383 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" event={"ID":"003e069b-84ac-4c68-9fa4-123d0a6a6417","Type":"ContainerDied","Data":"57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6"} Nov 02 14:06:57.559533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.559426 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c" event={"ID":"003e069b-84ac-4c68-9fa4-123d0a6a6417","Type":"ContainerDied","Data":"c3fdb1baea2d0c5495359357bc9fe11788616bedb86a4c9d3ab38a32c7463ec7"} Nov 02 14:06:57.568387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.568362 5591 scope.go:122] "RemoveContainer" containerID="c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6" Nov 02 14:06:57.568679 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:57.568656 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6\": container with ID starting with c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6 not found: ID does not exist" containerID="c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6" Nov 02 14:06:57.568754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.568686 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6"} err="failed to get container status \"c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6\": rpc error: code = NotFound desc = could not find container \"c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6\": container with ID starting with c693a0083c998bd647c7c4a2db77bf79d0a4d8d26a72e6a68b92cf8fc643a9f6 not found: ID does not exist" Nov 02 14:06:57.568754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.568705 5591 scope.go:122] "RemoveContainer" containerID="57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6" Nov 02 14:06:57.579767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.579745 5591 scope.go:122] "RemoveContainer" containerID="57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6" Nov 02 14:06:57.580154 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:06:57.580121 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6\": container with ID starting with 57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6 not found: ID does not exist" containerID="57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6" Nov 02 14:06:57.580217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.580157 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6"} err="failed to get container status \"57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6\": rpc error: code = NotFound desc = could not find container \"57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6\": container with ID starting with 57e186c9a068c178710b8ef0f8b9de779bc078a2eb7a262416dc282ecd7b79a6 not found: ID does not exist" Nov 02 14:06:57.595861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.595608 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-kube-api-access-nqcjg\" (UniqueName: \"kubernetes.io/projected/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-kube-api-access-nqcjg\") pod \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\" (UID: \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\") " Nov 02 14:06:57.595861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.595694 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003e069b-84ac-4c68-9fa4-123d0a6a6417-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e069b-84ac-4c68-9fa4-123d0a6a6417-plugins-config-volume\") pod \"003e069b-84ac-4c68-9fa4-123d0a6a6417\" (UID: \"003e069b-84ac-4c68-9fa4-123d0a6a6417\") " Nov 02 14:06:57.596238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.595951 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-plugins-config-volume\") pod \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\" (UID: \"e59169b9-f374-4c2c-9396-95c7b4cb3a9f\") " Nov 02 14:06:57.596238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.596033 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003e069b-84ac-4c68-9fa4-123d0a6a6417-kube-api-access-b6jnb\" (UniqueName: \"kubernetes.io/projected/003e069b-84ac-4c68-9fa4-123d0a6a6417-kube-api-access-b6jnb\") pod \"003e069b-84ac-4c68-9fa4-123d0a6a6417\" (UID: \"003e069b-84ac-4c68-9fa4-123d0a6a6417\") " Nov 02 14:06:57.596238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.596092 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003e069b-84ac-4c68-9fa4-123d0a6a6417-plugins-config-volume" pod "003e069b-84ac-4c68-9fa4-123d0a6a6417" (UID: "003e069b-84ac-4c68-9fa4-123d0a6a6417"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:06:57.596797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.596482 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-plugins-config-volume" pod "e59169b9-f374-4c2c-9396-95c7b4cb3a9f" (UID: "e59169b9-f374-4c2c-9396-95c7b4cb3a9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:06:57.596797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.596783 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003e069b-84ac-4c68-9fa4-123d0a6a6417-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:57.596961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.596806 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:57.598818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.598793 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003e069b-84ac-4c68-9fa4-123d0a6a6417-kube-api-access-b6jnb" pod "003e069b-84ac-4c68-9fa4-123d0a6a6417" (UID: "003e069b-84ac-4c68-9fa4-123d0a6a6417"). InnerVolumeSpecName "kube-api-access-b6jnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:57.599830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.599810 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-kube-api-access-nqcjg" pod "e59169b9-f374-4c2c-9396-95c7b4cb3a9f" (UID: "e59169b9-f374-4c2c-9396-95c7b4cb3a9f"). InnerVolumeSpecName "kube-api-access-nqcjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:57.698468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.698433 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jnb\" (UniqueName: \"kubernetes.io/projected/003e069b-84ac-4c68-9fa4-123d0a6a6417-kube-api-access-b6jnb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:57.698468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.698467 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqcjg\" (UniqueName: \"kubernetes.io/projected/e59169b9-f374-4c2c-9396-95c7b4cb3a9f-kube-api-access-nqcjg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:06:57.914825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.914786 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr"] Nov 02 14:06:57.926838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.926807 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-9c6fd5db7-b9qqr"] Nov 02 14:06:57.955546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.955515 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c"] Nov 02 14:06:57.958334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:57.958312 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-d6fdffdc5-cgr9c"] Nov 02 14:06:59.010157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:59.006596 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:06:59.433252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:59.433218 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003e069b-84ac-4c68-9fa4-123d0a6a6417" path="/var/lib/kubelet/pods/003e069b-84ac-4c68-9fa4-123d0a6a6417/volumes" Nov 02 14:06:59.433665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:06:59.433649 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59169b9-f374-4c2c-9396-95c7b4cb3a9f" path="/var/lib/kubelet/pods/e59169b9-f374-4c2c-9396-95c7b4cb3a9f/volumes" Nov 02 14:07:00.935021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:00.934965 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:01.074503 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:01.074458 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:07:01.210060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:01.208862 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:01.939153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:01.939124 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 14:07:01.960485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:01.960455 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 14:07:01.964172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:01.964150 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065460 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-istio-token\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-istio-token\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065524 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-kube-api-access-q4679\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-kube-api-access-q4679\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065599 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-socket\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065666 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-certs\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065707 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-podinfo\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065768 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74abb745-c01a-4429-b857-9d4cfd7f2b95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74abb745-c01a-4429-b857-9d4cfd7f2b95-istiod-ca-cert\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065803 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-socket\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065864 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-credential-socket\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065894 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-istio-token\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-istio-token\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065930 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4251524d-6cb3-430f-930f-02bc806b91f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4251524d-6cb3-430f-930f-02bc806b91f9-istiod-ca-cert\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.065959 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-credential-socket\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066056 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-data\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066091 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-data\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066149 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-socket\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066177 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-kube-api-access-vxjll\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-kube-api-access-vxjll\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066210 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-kube-api-access-gz42l\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-kube-api-access-gz42l\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066258 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-data\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066290 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-envoy\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066325 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-envoy\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066386 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-envoy\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066422 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4251524d-6cb3-430f-930f-02bc806b91f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4251524d-6cb3-430f-930f-02bc806b91f9-istio-podinfo\") pod \"4251524d-6cb3-430f-930f-02bc806b91f9\" (UID: \"4251524d-6cb3-430f-930f-02bc806b91f9\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066535 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7a6917-c963-471b-8f38-ad047570afb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a6917-c963-471b-8f38-ad047570afb1-istiod-ca-cert\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066595 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-certs\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066706 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-token\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-token\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066744 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b7a6917-c963-471b-8f38-ad047570afb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7a6917-c963-471b-8f38-ad047570afb1-istio-podinfo\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066791 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-credential-socket\") pod \"74abb745-c01a-4429-b857-9d4cfd7f2b95\" (UID: \"74abb745-c01a-4429-b857-9d4cfd7f2b95\") " Nov 02 14:07:02.084829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.066842 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-certs\") pod \"6b7a6917-c963-471b-8f38-ad047570afb1\" (UID: \"6b7a6917-c963-471b-8f38-ad047570afb1\") " Nov 02 14:07:02.108009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.101086 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-socket" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.111438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.111109 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-data" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.111438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.111384 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-certs" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.145189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.130257 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4251524d-6cb3-430f-930f-02bc806b91f9-istiod-ca-cert" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:02.145189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.132065 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-socket" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.145189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.141525 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-credential-socket" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.145189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.141860 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-data" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.145189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.143580 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-credential-socket" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.150208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.148894 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-data" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.150208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.149395 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-socket" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.150208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.149603 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-certs" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.150208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.149876 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7a6917-c963-471b-8f38-ad047570afb1-istiod-ca-cert" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:02.151869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.151810 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-certs" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.161456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.160060 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74abb745-c01a-4429-b857-9d4cfd7f2b95-istiod-ca-cert" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:02.162164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.162013 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-istio-token" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:02.162330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.162096 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-credential-socket" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.162544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.162512 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-envoy" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170016 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170049 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170063 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170080 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170093 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74abb745-c01a-4429-b857-9d4cfd7f2b95-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170106 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170118 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170134 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170147 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4251524d-6cb3-430f-930f-02bc806b91f9-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170159 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170174 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170190 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170202 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170215 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170228 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4251524d-6cb3-430f-930f-02bc806b91f9-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170283 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a6917-c963-471b-8f38-ad047570afb1-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.171095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.170299 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.217006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.212022 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-token" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:02.234243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.224659 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4251524d-6cb3-430f-930f-02bc806b91f9-istio-podinfo" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:07:02.234243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.229496 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b7a6917-c963-471b-8f38-ad047570afb1-istio-podinfo" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:07:02.234243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.234191 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-envoy" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.234900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.234819 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-istio-token" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:02.235242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.235195 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-kube-api-access-vxjll" pod "4251524d-6cb3-430f-930f-02bc806b91f9" (UID: "4251524d-6cb3-430f-930f-02bc806b91f9"). InnerVolumeSpecName "kube-api-access-vxjll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:02.240538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.240494 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-envoy" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:02.241125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.241086 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-podinfo" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:07:02.244560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.244522 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-kube-api-access-gz42l" pod "6b7a6917-c963-471b-8f38-ad047570afb1" (UID: "6b7a6917-c963-471b-8f38-ad047570afb1"). InnerVolumeSpecName "kube-api-access-gz42l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:02.250275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.250243 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-kube-api-access-q4679" pod "74abb745-c01a-4429-b857-9d4cfd7f2b95" (UID: "74abb745-c01a-4429-b857-9d4cfd7f2b95"). InnerVolumeSpecName "kube-api-access-q4679". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:02.272364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272323 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxjll\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-kube-api-access-vxjll\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272365 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz42l\" (UniqueName: \"kubernetes.io/projected/6b7a6917-c963-471b-8f38-ad047570afb1-kube-api-access-gz42l\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272382 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7a6917-c963-471b-8f38-ad047570afb1-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272396 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272409 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4251524d-6cb3-430f-930f-02bc806b91f9-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272421 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272434 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7a6917-c963-471b-8f38-ad047570afb1-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272447 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4251524d-6cb3-430f-930f-02bc806b91f9-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272464 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4679\" (UniqueName: \"kubernetes.io/projected/74abb745-c01a-4429-b857-9d4cfd7f2b95-kube-api-access-q4679\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.272616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.272477 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74abb745-c01a-4429-b857-9d4cfd7f2b95-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:02.595883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.595791 5591 generic.go:356] "Generic (PLEG): container finished" podID="4251524d-6cb3-430f-930f-02bc806b91f9" containerID="e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7" exitCode=0 Nov 02 14:07:02.595883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.595843 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" Nov 02 14:07:02.595883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.595871 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" event={"ID":"4251524d-6cb3-430f-930f-02bc806b91f9","Type":"ContainerDied","Data":"e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7"} Nov 02 14:07:02.596150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.595936 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv" event={"ID":"4251524d-6cb3-430f-930f-02bc806b91f9","Type":"ContainerDied","Data":"384788ac1a6677f3a0387d9f803e8409d9deb4c8ab4ccd5b1113e0fe236e5c9d"} Nov 02 14:07:02.596150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.595970 5591 scope.go:122] "RemoveContainer" containerID="e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7" Nov 02 14:07:02.597357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.597332 5591 generic.go:356] "Generic (PLEG): container finished" podID="6b7a6917-c963-471b-8f38-ad047570afb1" containerID="46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15" exitCode=0 Nov 02 14:07:02.597478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.597388 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" Nov 02 14:07:02.597478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.597419 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" event={"ID":"6b7a6917-c963-471b-8f38-ad047570afb1","Type":"ContainerDied","Data":"46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15"} Nov 02 14:07:02.597478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.597472 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9" event={"ID":"6b7a6917-c963-471b-8f38-ad047570afb1","Type":"ContainerDied","Data":"0bffbcbd26ec72cef5e08722b127e66589acf4dacf6bfa6bcd1d7442c65518d5"} Nov 02 14:07:02.599161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.599136 5591 generic.go:356] "Generic (PLEG): container finished" podID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerID="a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036" exitCode=0 Nov 02 14:07:02.599282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.599183 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" Nov 02 14:07:02.599282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.599184 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" event={"ID":"74abb745-c01a-4429-b857-9d4cfd7f2b95","Type":"ContainerDied","Data":"a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036"} Nov 02 14:07:02.599383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.599299 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4" event={"ID":"74abb745-c01a-4429-b857-9d4cfd7f2b95","Type":"ContainerDied","Data":"5cde715fce2e8ef3b7d5cc022b074dc8d6b0b5cf7ce6b68145a204072cefb379"} Nov 02 14:07:02.608187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.607917 5591 scope.go:122] "RemoveContainer" containerID="e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7" Nov 02 14:07:02.608287 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:02.608237 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7\": container with ID starting with e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7 not found: ID does not exist" containerID="e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7" Nov 02 14:07:02.608287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.608271 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7"} err="failed to get container status \"e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7\": rpc error: code = NotFound desc = could not find container \"e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7\": container with ID starting with e10e73e772571e94c25e89d14c384c0f2e6cd422223250c259ea697babdf3bc7 not found: ID does not exist" Nov 02 14:07:02.608416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.608296 5591 scope.go:122] "RemoveContainer" containerID="46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15" Nov 02 14:07:02.621364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.619562 5591 scope.go:122] "RemoveContainer" containerID="46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15" Nov 02 14:07:02.621364 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:02.620085 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15\": container with ID starting with 46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15 not found: ID does not exist" containerID="46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15" Nov 02 14:07:02.621364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.620115 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15"} err="failed to get container status \"46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15\": rpc error: code = NotFound desc = could not find container \"46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15\": container with ID starting with 46f16a8cccf9f8eb871d4e85e8f0d4d0254d5e05221afb6472d364101ac1cb15 not found: ID does not exist" Nov 02 14:07:02.621364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.620135 5591 scope.go:122] "RemoveContainer" containerID="a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036" Nov 02 14:07:02.630340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.630320 5591 scope.go:122] "RemoveContainer" containerID="a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036" Nov 02 14:07:02.630600 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:02.630583 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036\": container with ID starting with a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036 not found: ID does not exist" containerID="a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036" Nov 02 14:07:02.630652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.630604 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036"} err="failed to get container status \"a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036\": rpc error: code = NotFound desc = could not find container \"a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036\": container with ID starting with a8b0d4d4d10542f201287b3ec58994c9bfa456382d5e64ae439b5d7d6e8d0036 not found: ID does not exist" Nov 02 14:07:02.656461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.656431 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4"] Nov 02 14:07:02.664384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.664358 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-w59w4"] Nov 02 14:07:02.707640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.707609 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9"] Nov 02 14:07:02.711384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.711364 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qp8z9"] Nov 02 14:07:02.753550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.753522 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv"] Nov 02 14:07:02.765401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:02.762134 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-dmgfv"] Nov 02 14:07:03.432778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:03.432742 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" path="/var/lib/kubelet/pods/4251524d-6cb3-430f-930f-02bc806b91f9/volumes" Nov 02 14:07:03.433313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:03.433296 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" path="/var/lib/kubelet/pods/6b7a6917-c963-471b-8f38-ad047570afb1/volumes" Nov 02 14:07:03.433742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:03.433729 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" path="/var/lib/kubelet/pods/74abb745-c01a-4429-b857-9d4cfd7f2b95/volumes" Nov 02 14:07:03.731080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:03.725363 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:04.933201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:04.933167 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:05.129531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.129425 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:05.287355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287273 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz"] Nov 02 14:07:05.287889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287875 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b" containerName="payload-processing" Nov 02 14:07:05.287929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287890 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b" containerName="payload-processing" Nov 02 14:07:05.287929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287906 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" Nov 02 14:07:05.287929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287913 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" Nov 02 14:07:05.287929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287923 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" Nov 02 14:07:05.287929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287928 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287943 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e59169b9-f374-4c2c-9396-95c7b4cb3a9f" containerName="payload-pre-processing" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287950 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59169b9-f374-4c2c-9396-95c7b4cb3a9f" containerName="payload-pre-processing" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287967 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7af67e4-3142-44e1-95cc-b69cbd3d46f1" containerName="payload-pre-processing" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287972 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7af67e4-3142-44e1-95cc-b69cbd3d46f1" containerName="payload-pre-processing" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.287996 5591 cpu_manager.go:414] "containerMap: removing container" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288004 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288027 5591 cpu_manager.go:414] "containerMap: removing container" podUID="003e069b-84ac-4c68-9fa4-123d0a6a6417" containerName="payload-processing" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288033 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003e069b-84ac-4c68-9fa4-123d0a6a6417" containerName="payload-processing" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288152 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7af67e4-3142-44e1-95cc-b69cbd3d46f1" containerName="payload-pre-processing" Nov 02 14:07:05.288161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288163 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="74abb745-c01a-4429-b857-9d4cfd7f2b95" containerName="istio-proxy" Nov 02 14:07:05.288451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288170 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="003e069b-84ac-4c68-9fa4-123d0a6a6417" containerName="payload-processing" Nov 02 14:07:05.288451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288179 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5d3ea4-0b9c-4fec-adb7-b06328b20f8b" containerName="payload-processing" Nov 02 14:07:05.288451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288185 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4251524d-6cb3-430f-930f-02bc806b91f9" containerName="istio-proxy" Nov 02 14:07:05.288451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288192 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7a6917-c963-471b-8f38-ad047570afb1" containerName="istio-proxy" Nov 02 14:07:05.288451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.288198 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59169b9-f374-4c2c-9396-95c7b4cb3a9f" containerName="payload-pre-processing" Nov 02 14:07:05.293508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.293491 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:05.323952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.323919 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60ad62-dockercfg-t4455\"" Nov 02 14:07:05.324613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.323954 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60ad62\"" Nov 02 14:07:05.379081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.379038 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz"] Nov 02 14:07:05.593027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.590216 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fce9f56-4586-479f-b01b-7adae0c869f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz\" (UID: \"4fce9f56-4586-479f-b01b-7adae0c869f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:05.593486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.593454 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75rs\" (UniqueName: \"kubernetes.io/projected/4fce9f56-4586-479f-b01b-7adae0c869f4-kube-api-access-q75rs\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz\" (UID: \"4fce9f56-4586-479f-b01b-7adae0c869f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:05.649896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.649860 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x"] Nov 02 14:07:05.654153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.654128 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x"] Nov 02 14:07:05.654292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.654227 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:05.694897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.694855 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fce9f56-4586-479f-b01b-7adae0c869f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz\" (UID: \"4fce9f56-4586-479f-b01b-7adae0c869f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:05.695174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.695033 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75rs\" (UniqueName: \"kubernetes.io/projected/4fce9f56-4586-479f-b01b-7adae0c869f4-kube-api-access-q75rs\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz\" (UID: \"4fce9f56-4586-479f-b01b-7adae0c869f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:05.695731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.695703 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fce9f56-4586-479f-b01b-7adae0c869f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz\" (UID: \"4fce9f56-4586-479f-b01b-7adae0c869f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:05.721368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.721346 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75rs\" (UniqueName: \"kubernetes.io/projected/4fce9f56-4586-479f-b01b-7adae0c869f4-kube-api-access-q75rs\") pod \"payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz\" (UID: \"4fce9f56-4586-479f-b01b-7adae0c869f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:05.796720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.796682 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x\" (UID: \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:05.796884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.796733 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8svp\" (UniqueName: \"kubernetes.io/projected/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-kube-api-access-n8svp\") pod \"payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x\" (UID: \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:05.898592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.898543 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x\" (UID: \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:05.898773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.898617 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8svp\" (UniqueName: \"kubernetes.io/projected/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-kube-api-access-n8svp\") pod \"payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x\" (UID: \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:05.899264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.899241 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x\" (UID: \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:05.907143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.907124 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:05.923282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.923256 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8svp\" (UniqueName: \"kubernetes.io/projected/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-kube-api-access-n8svp\") pod \"payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x\" (UID: \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:05.968360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:05.968332 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:06.106241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.095749 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:06.224359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.224331 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz"] Nov 02 14:07:06.330206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.330172 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x"] Nov 02 14:07:06.330323 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:07:06.330218 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6caca4_7449_4a83_b2b1_7747ff0e2fc9.slice/crio-4c97db9d277be2b9d6740ad17c3433032313b217ae37fa5c5148eb9233356348 WatchSource:0}: Error finding container 4c97db9d277be2b9d6740ad17c3433032313b217ae37fa5c5148eb9233356348: Status 404 returned error can't find the container with id 4c97db9d277be2b9d6740ad17c3433032313b217ae37fa5c5148eb9233356348 Nov 02 14:07:06.657502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.657462 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" event={"ID":"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9","Type":"ContainerStarted","Data":"8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f"} Nov 02 14:07:06.657682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.657527 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" event={"ID":"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9","Type":"ContainerStarted","Data":"4c97db9d277be2b9d6740ad17c3433032313b217ae37fa5c5148eb9233356348"} Nov 02 14:07:06.657682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.657604 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:06.659351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.659323 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" event={"ID":"4fce9f56-4586-479f-b01b-7adae0c869f4","Type":"ContainerStarted","Data":"1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8"} Nov 02 14:07:06.659512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.659367 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" event={"ID":"4fce9f56-4586-479f-b01b-7adae0c869f4","Type":"ContainerStarted","Data":"7ce890dece67431765f880604a7fb10cc32aed5e987011006d04e59839bda430"} Nov 02 14:07:06.659512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.659419 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:06.775237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.775181 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" podStartSLOduration=1.775167954 podStartE2EDuration="1.775167954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:06.773478546 +0000 UTC m=+1375.499100316" watchObservedRunningTime="2026-11-02 14:07:06.775167954 +0000 UTC m=+1375.500789771" Nov 02 14:07:06.776242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:06.776210 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" podStartSLOduration=1.776202922 podStartE2EDuration="1.776202922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:06.729283678 +0000 UTC m=+1375.454905448" watchObservedRunningTime="2026-11-02 14:07:06.776202922 +0000 UTC m=+1375.501824692" Nov 02 14:07:07.436402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:07.436362 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:07.799081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:07.798967 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:09.094995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.094955 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f"] Nov 02 14:07:09.099356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.099336 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.122199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.122153 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f"] Nov 02 14:07:09.131752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.131711 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f66ca3-openshift-default-dockercfg-z7wqz\"" Nov 02 14:07:09.257597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.257557 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.258181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.258133 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.258477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.258439 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.258678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.258661 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.258890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.258867 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484t2\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-kube-api-access-484t2\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.259166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.259136 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.279037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.277294 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.279037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.277423 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55839559-f783-40f1-b436-3e4af057ee6a-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.279037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.277501 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55839559-f783-40f1-b436-3e4af057ee6a-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.388924 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.389159 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.389243 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.389279 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.389314 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484t2\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-kube-api-access-484t2\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.389404 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.390513 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.391100 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.391741 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.392420 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.392501 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55839559-f783-40f1-b436-3e4af057ee6a-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.392581 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55839559-f783-40f1-b436-3e4af057ee6a-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.393334 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55839559-f783-40f1-b436-3e4af057ee6a-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.394005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.393561 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.397261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.397234 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.397858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.397835 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55839559-f783-40f1-b436-3e4af057ee6a-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.415272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.415207 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.415675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.415647 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-484t2\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-kube-api-access-484t2\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:09.614343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.614292 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:09.714196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:09.714107 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:10.045096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:10.045061 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f"] Nov 02 14:07:10.045974 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:07:10.045947 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55839559_f783_40f1_b436_3e4af057ee6a.slice/crio-039966c0df921bc44dcd3b450a9eb87212786a2edf4bbacdb0a73d1181e4ae1a WatchSource:0}: Error finding container 039966c0df921bc44dcd3b450a9eb87212786a2edf4bbacdb0a73d1181e4ae1a: Status 404 returned error can't find the container with id 039966c0df921bc44dcd3b450a9eb87212786a2edf4bbacdb0a73d1181e4ae1a Nov 02 14:07:10.049060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:10.049027 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:07:10.049177 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:10.049113 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:07:10.049177 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:10.049159 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:07:10.689967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:10.689933 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" event={"ID":"55839559-f783-40f1-b436-3e4af057ee6a","Type":"ContainerStarted","Data":"a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a"} Nov 02 14:07:10.689967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:10.690010 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" event={"ID":"55839559-f783-40f1-b436-3e4af057ee6a","Type":"ContainerStarted","Data":"039966c0df921bc44dcd3b450a9eb87212786a2edf4bbacdb0a73d1181e4ae1a"} Nov 02 14:07:10.714683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:10.714652 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:10.766341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:10.766281 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" podStartSLOduration=1.7662670249999999 podStartE2EDuration="1.766267025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:10.750640532 +0000 UTC m=+1379.476262303" watchObservedRunningTime="2026-11-02 14:07:10.766267025 +0000 UTC m=+1379.491888798" Nov 02 14:07:11.713388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:11.711756 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:11.753011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:11.748147 5591 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:07:11.753011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:11.748215 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:07:12.714779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:12.714743 5591 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:07:12.715209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:12.714797 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:07:13.715237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:13.715196 5591 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:07:13.715633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:13.715298 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:07:14.350053 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:14.349501 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:14.519324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:14.519254 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:14.715065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:14.715028 5591 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:07:14.715251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:14.715118 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:07:15.715282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:15.715239 5591 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:07:15.715661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:15.715298 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:07:16.239004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:16.233292 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:16.720398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:16.720368 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:16.720877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:16.720865 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:16.721730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:16.721710 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:07:16.789384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:16.789333 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:17.672445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:17.672417 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:07:17.672685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:17.672670 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:07:18.370010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:18.368976 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:18.522573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:18.522536 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:07:19.765052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:19.757189 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:19.982141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:19.982111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:20.217008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:20.214478 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:20.958698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:20.958660 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:22.139389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:22.139355 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w"] Nov 02 14:07:24.595571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:24.595534 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:24.827564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:24.827534 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:25.028368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:25.028324 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:25.352340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:25.352256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:26.187939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:26.187163 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:26.977489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:26.977456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:27.140250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.140177 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" podUID="0cef97ab-3764-4163-908d-4ef66716b0aa" containerName="payload-pre-processing" containerID="cri-o://8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9" gracePeriod=25 Nov 02 14:07:27.418888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.418859 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:07:27.465321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.465287 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cef97ab-3764-4163-908d-4ef66716b0aa-kube-api-access-jrmsm\" (UniqueName: \"kubernetes.io/projected/0cef97ab-3764-4163-908d-4ef66716b0aa-kube-api-access-jrmsm\") pod \"0cef97ab-3764-4163-908d-4ef66716b0aa\" (UID: \"0cef97ab-3764-4163-908d-4ef66716b0aa\") " Nov 02 14:07:27.465515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.465414 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cef97ab-3764-4163-908d-4ef66716b0aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cef97ab-3764-4163-908d-4ef66716b0aa-plugins-config-volume\") pod \"0cef97ab-3764-4163-908d-4ef66716b0aa\" (UID: \"0cef97ab-3764-4163-908d-4ef66716b0aa\") " Nov 02 14:07:27.465795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.465770 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cef97ab-3764-4163-908d-4ef66716b0aa-plugins-config-volume" pod "0cef97ab-3764-4163-908d-4ef66716b0aa" (UID: "0cef97ab-3764-4163-908d-4ef66716b0aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:27.466111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.466093 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cef97ab-3764-4163-908d-4ef66716b0aa-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:27.467625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.467601 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cef97ab-3764-4163-908d-4ef66716b0aa-kube-api-access-jrmsm" pod "0cef97ab-3764-4163-908d-4ef66716b0aa" (UID: "0cef97ab-3764-4163-908d-4ef66716b0aa"). InnerVolumeSpecName "kube-api-access-jrmsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:27.568514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.568437 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrmsm\" (UniqueName: \"kubernetes.io/projected/0cef97ab-3764-4163-908d-4ef66716b0aa-kube-api-access-jrmsm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:27.915803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.915769 5591 generic.go:356] "Generic (PLEG): container finished" podID="0cef97ab-3764-4163-908d-4ef66716b0aa" containerID="8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9" exitCode=0 Nov 02 14:07:27.916032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.915821 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" Nov 02 14:07:27.916032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.915857 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" event={"ID":"0cef97ab-3764-4163-908d-4ef66716b0aa","Type":"ContainerDied","Data":"8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9"} Nov 02 14:07:27.916032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.915916 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w" event={"ID":"0cef97ab-3764-4163-908d-4ef66716b0aa","Type":"ContainerDied","Data":"5bab9bfd2f9fbfc12126078aa489e505cf32621cba88bb04451d358dee08cfb8"} Nov 02 14:07:27.916032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.915939 5591 scope.go:122] "RemoveContainer" containerID="8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9" Nov 02 14:07:27.926320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.926301 5591 scope.go:122] "RemoveContainer" containerID="8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9" Nov 02 14:07:27.926591 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:27.926573 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9\": container with ID starting with 8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9 not found: ID does not exist" containerID="8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9" Nov 02 14:07:27.926646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.926600 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9"} err="failed to get container status \"8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9\": rpc error: code = NotFound desc = could not find container \"8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9\": container with ID starting with 8b105e125517025c6a7585d8eff81ec274fd1cdd9381518b53684080438158a9 not found: ID does not exist" Nov 02 14:07:27.973489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.973454 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w"] Nov 02 14:07:27.986449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:27.986416 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7d2df-b5b7cb559-sj69w"] Nov 02 14:07:28.075046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:28.068786 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:29.215869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:29.215824 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm"] Nov 02 14:07:29.346686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:29.346532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:29.457010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:29.452970 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cef97ab-3764-4163-908d-4ef66716b0aa" path="/var/lib/kubelet/pods/0cef97ab-3764-4163-908d-4ef66716b0aa/volumes" Nov 02 14:07:29.528347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:29.518616 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:30.073844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:30.073795 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:31.074044 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:31.074008 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:07:31.915606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:31.915572 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:32.128466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:32.128399 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:32.501010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:32.500118 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:32.732498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:32.732458 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:32.954219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:32.954182 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:33.233183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:33.233101 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc"] Nov 02 14:07:33.251509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:33.250744 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" containerID="cri-o://537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2" gracePeriod=30 Nov 02 14:07:33.288587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:33.288556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:33.722090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:33.719625 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:34.198311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.198275 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:34.217277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.217224 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" podUID="c75085e6-f3ae-403e-a714-095deb1ea874" containerName="payload-processing" containerID="cri-o://7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292" gracePeriod=25 Nov 02 14:07:34.515830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.515804 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:07:34.695344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.695308 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75085e6-f3ae-403e-a714-095deb1ea874-kube-api-access-wppsz\" (UniqueName: \"kubernetes.io/projected/c75085e6-f3ae-403e-a714-095deb1ea874-kube-api-access-wppsz\") pod \"c75085e6-f3ae-403e-a714-095deb1ea874\" (UID: \"c75085e6-f3ae-403e-a714-095deb1ea874\") " Nov 02 14:07:34.695542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.695470 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75085e6-f3ae-403e-a714-095deb1ea874-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75085e6-f3ae-403e-a714-095deb1ea874-plugins-config-volume\") pod \"c75085e6-f3ae-403e-a714-095deb1ea874\" (UID: \"c75085e6-f3ae-403e-a714-095deb1ea874\") " Nov 02 14:07:34.695841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.695816 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75085e6-f3ae-403e-a714-095deb1ea874-plugins-config-volume" pod "c75085e6-f3ae-403e-a714-095deb1ea874" (UID: "c75085e6-f3ae-403e-a714-095deb1ea874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:34.695964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.695949 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75085e6-f3ae-403e-a714-095deb1ea874-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:34.697442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.697420 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75085e6-f3ae-403e-a714-095deb1ea874-kube-api-access-wppsz" pod "c75085e6-f3ae-403e-a714-095deb1ea874" (UID: "c75085e6-f3ae-403e-a714-095deb1ea874"). InnerVolumeSpecName "kube-api-access-wppsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:34.797214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:34.797121 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wppsz\" (UniqueName: \"kubernetes.io/projected/c75085e6-f3ae-403e-a714-095deb1ea874-kube-api-access-wppsz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:35.263850 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.263814 5591 generic.go:356] "Generic (PLEG): container finished" podID="c75085e6-f3ae-403e-a714-095deb1ea874" containerID="7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292" exitCode=0 Nov 02 14:07:35.264086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.263869 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" Nov 02 14:07:35.264086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.263901 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" event={"ID":"c75085e6-f3ae-403e-a714-095deb1ea874","Type":"ContainerDied","Data":"7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292"} Nov 02 14:07:35.264086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.263959 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm" event={"ID":"c75085e6-f3ae-403e-a714-095deb1ea874","Type":"ContainerDied","Data":"000ba9e9c9c56955b1601259e54ff7c9e010080bdc2d523c7091d2477cbb439a"} Nov 02 14:07:35.264086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.264008 5591 scope.go:122] "RemoveContainer" containerID="7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292" Nov 02 14:07:35.273994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.273962 5591 scope.go:122] "RemoveContainer" containerID="7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292" Nov 02 14:07:35.274284 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:35.274257 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292\": container with ID starting with 7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292 not found: ID does not exist" containerID="7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292" Nov 02 14:07:35.274336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.274293 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292"} err="failed to get container status \"7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292\": rpc error: code = NotFound desc = could not find container \"7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292\": container with ID starting with 7276038a2d60673988a41e42d87d3bbaa5385c91bf8b2537c8e47ae3b9da4292 not found: ID does not exist" Nov 02 14:07:35.322904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.322867 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm"] Nov 02 14:07:35.335780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.335742 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-c594c76f9-c82dm"] Nov 02 14:07:35.431262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:35.431221 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75085e6-f3ae-403e-a714-095deb1ea874" path="/var/lib/kubelet/pods/c75085e6-f3ae-403e-a714-095deb1ea874/volumes" Nov 02 14:07:36.342069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:36.339427 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:38.324818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.324778 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:38.736102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.736074 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:07:38.769961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.769925 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-envoy\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.769970 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33cb9750-f264-4b5f-8069-8af98fdc5001-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cb9750-f264-4b5f-8069-8af98fdc5001-istiod-ca-cert\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770025 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-token\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-token\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770081 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-kube-api-access-q4h4q\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-kube-api-access-q4h4q\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770108 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-credential-socket\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770137 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-data\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770239 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-certs\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770264 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-socket\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770436 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-podinfo\") pod \"33cb9750-f264-4b5f-8069-8af98fdc5001\" (UID: \"33cb9750-f264-4b5f-8069-8af98fdc5001\") " Nov 02 14:07:38.770657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770524 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33cb9750-f264-4b5f-8069-8af98fdc5001-istiod-ca-cert" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:38.770657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.770640 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-certs" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:38.771378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.771145 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-credential-socket" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:38.771378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.771145 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-data" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:38.771378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.771181 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-socket" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:38.771378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.771314 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33cb9750-f264-4b5f-8069-8af98fdc5001-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:38.771378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.771329 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:38.771378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.771340 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:38.771378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.771349 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:38.771378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.771357 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:38.772899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.772873 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-podinfo" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:07:38.773108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.773085 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-envoy" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:38.773246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.773225 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-kube-api-access-q4h4q" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "kube-api-access-q4h4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:38.773373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.773359 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-token" pod "33cb9750-f264-4b5f-8069-8af98fdc5001" (UID: "33cb9750-f264-4b5f-8069-8af98fdc5001"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:38.872843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.872800 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4h4q\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-kube-api-access-q4h4q\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:38.872843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.872836 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:38.872843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.872850 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:38.873144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:38.872862 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33cb9750-f264-4b5f-8069-8af98fdc5001-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:39.360265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.360231 5591 generic.go:356] "Generic (PLEG): container finished" podID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerID="537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2" exitCode=0 Nov 02 14:07:39.360834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.360281 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" Nov 02 14:07:39.360834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.360301 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" event={"ID":"33cb9750-f264-4b5f-8069-8af98fdc5001","Type":"ContainerDied","Data":"537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2"} Nov 02 14:07:39.360834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.360351 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc" event={"ID":"33cb9750-f264-4b5f-8069-8af98fdc5001","Type":"ContainerDied","Data":"ad449f23b42dffaf6aefdf0316a92852bfaace53b5ce7148105ffbef91a49e91"} Nov 02 14:07:39.360834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.360374 5591 scope.go:122] "RemoveContainer" containerID="537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2" Nov 02 14:07:39.361374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.361338 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:39.372390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.372233 5591 scope.go:122] "RemoveContainer" containerID="39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2" Nov 02 14:07:39.381239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.381213 5591 scope.go:122] "RemoveContainer" containerID="537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2" Nov 02 14:07:39.381509 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:39.381489 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2\": container with ID starting with 537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2 not found: ID does not exist" containerID="537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2" Nov 02 14:07:39.381588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.381524 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2"} err="failed to get container status \"537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2\": rpc error: code = NotFound desc = could not find container \"537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2\": container with ID starting with 537f6492d2da66c86c0318e9bec503521e7a32a70f044b1d92712171a15471c2 not found: ID does not exist" Nov 02 14:07:39.381588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.381550 5591 scope.go:122] "RemoveContainer" containerID="39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2" Nov 02 14:07:39.381786 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:39.381768 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2\": container with ID starting with 39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2 not found: ID does not exist" containerID="39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2" Nov 02 14:07:39.381829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.381791 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2"} err="failed to get container status \"39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2\": rpc error: code = NotFound desc = could not find container \"39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2\": container with ID starting with 39735b52ada90c912437e1e0d47ef984e480dc5d3af68b64fb525ba6544579f2 not found: ID does not exist" Nov 02 14:07:39.434782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.434752 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc"] Nov 02 14:07:39.442268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.442238 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c7d2df-openshift-default-5f757b6777-r5ccc"] Nov 02 14:07:39.814171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.814138 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq"] Nov 02 14:07:39.814889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.814864 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c75085e6-f3ae-403e-a714-095deb1ea874" containerName="payload-processing" Nov 02 14:07:39.814889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.814885 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75085e6-f3ae-403e-a714-095deb1ea874" containerName="payload-processing" Nov 02 14:07:39.815112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.814914 5591 cpu_manager.go:414] "containerMap: removing container" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" Nov 02 14:07:39.815112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.814920 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" Nov 02 14:07:39.815112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.814950 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0cef97ab-3764-4163-908d-4ef66716b0aa" containerName="payload-pre-processing" Nov 02 14:07:39.815112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.814957 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cef97ab-3764-4163-908d-4ef66716b0aa" containerName="payload-pre-processing" Nov 02 14:07:39.815112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.815069 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" Nov 02 14:07:39.815112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.815083 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cef97ab-3764-4163-908d-4ef66716b0aa" containerName="payload-pre-processing" Nov 02 14:07:39.815112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.815091 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75085e6-f3ae-403e-a714-095deb1ea874" containerName="payload-processing" Nov 02 14:07:39.820059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.820039 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.848486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.848452 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-54f2fe-openshift-default-dockercfg-w7q8j\"" Nov 02 14:07:39.851794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.851765 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq"] Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910044 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtj4d\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-kube-api-access-xtj4d\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910108 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea174eb3-9111-4203-ac02-d7156399a40f-istiod-ca-cert\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910206 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-istio-token\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910280 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-credential-socket\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910368 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-envoy\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910441 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-socket\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910485 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea174eb3-9111-4203-ac02-d7156399a40f-istio-podinfo\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910511 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-certs\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.922008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.910560 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-data\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:39.964405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:39.964369 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:40.030729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.030689 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-socket\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.033041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.031889 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-socket\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.033815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.033786 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea174eb3-9111-4203-ac02-d7156399a40f-istio-podinfo\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.035357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.033965 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-certs\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.035599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.035576 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-data\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.035798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.035782 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtj4d\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-kube-api-access-xtj4d\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.035924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.035910 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea174eb3-9111-4203-ac02-d7156399a40f-istiod-ca-cert\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.036145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.036126 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-istio-token\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.078699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.078620 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-credential-socket\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.094010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.036536 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-data\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.094010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.037712 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea174eb3-9111-4203-ac02-d7156399a40f-istiod-ca-cert\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.094010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.079025 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-envoy\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.094010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.079714 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-credential-socket\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.094010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.036760 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-certs\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.094010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.080388 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea174eb3-9111-4203-ac02-d7156399a40f-istio-podinfo\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.094010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.092190 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-envoy\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.101007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.100939 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtj4d\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-kube-api-access-xtj4d\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.107016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.101682 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-istio-token\") pod \"e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.133299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.133262 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:40.415338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.415305 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:40.488382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.488347 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq"] Nov 02 14:07:40.489641 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:07:40.489610 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea174eb3_9111_4203_ac02_d7156399a40f.slice/crio-f5072b705ce74b35ab76102b96a4a1a2ac5cc73bdaec9d83dce6b5f98e719ad7 WatchSource:0}: Error finding container f5072b705ce74b35ab76102b96a4a1a2ac5cc73bdaec9d83dce6b5f98e719ad7: Status 404 returned error can't find the container with id f5072b705ce74b35ab76102b96a4a1a2ac5cc73bdaec9d83dce6b5f98e719ad7 Nov 02 14:07:40.492636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.492601 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:07:40.492726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.492668 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:07:40.492726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.492698 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:07:40.596926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.596847 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:40.926013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:40.924116 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:41.107761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:41.107714 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:41.533005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:41.514330 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" path="/var/lib/kubelet/pods/33cb9750-f264-4b5f-8069-8af98fdc5001/volumes" Nov 02 14:07:41.533005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:41.514823 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" event={"ID":"ea174eb3-9111-4203-ac02-d7156399a40f","Type":"ContainerStarted","Data":"69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd"} Nov 02 14:07:41.533005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:41.514879 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" event={"ID":"ea174eb3-9111-4203-ac02-d7156399a40f","Type":"ContainerStarted","Data":"f5072b705ce74b35ab76102b96a4a1a2ac5cc73bdaec9d83dce6b5f98e719ad7"} Nov 02 14:07:42.134211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:42.134179 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:42.139649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:42.139622 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:42.207569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:42.207518 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" podStartSLOduration=3.207504692 podStartE2EDuration="3.207504692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:41.561693078 +0000 UTC m=+1410.287314850" watchObservedRunningTime="2026-11-02 14:07:42.207504692 +0000 UTC m=+1410.933126532" Nov 02 14:07:42.391932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:42.391839 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:42.513025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:42.512961 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:42.513898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:42.513879 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:07:42.562286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:42.562239 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:44.047285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:44.047252 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:44.252384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:44.252354 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:44.970790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:44.970750 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:45.759514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.759474 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:45.847649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.847620 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr"] Nov 02 14:07:45.848246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.848231 5591 cpu_manager.go:414] "containerMap: removing container" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" Nov 02 14:07:45.848298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.848247 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" Nov 02 14:07:45.848378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.848369 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="33cb9750-f264-4b5f-8069-8af98fdc5001" containerName="istio-proxy" Nov 02 14:07:45.853034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.853016 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:45.887066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.887041 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-54f2fe-dockercfg-4j9xr\"" Nov 02 14:07:45.894723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.894694 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809a5f84-4c45-48d1-81b1-926f8e250dba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr\" (UID: \"809a5f84-4c45-48d1-81b1-926f8e250dba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:45.894927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.894908 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph6xh\" (UniqueName: \"kubernetes.io/projected/809a5f84-4c45-48d1-81b1-926f8e250dba-kube-api-access-ph6xh\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr\" (UID: \"809a5f84-4c45-48d1-81b1-926f8e250dba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:45.909655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.909626 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr"] Nov 02 14:07:45.929797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.929763 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv"] Nov 02 14:07:45.934457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.934433 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:45.955886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.955857 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-54f2fe\"" Nov 02 14:07:45.964300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:45.964265 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv"] Nov 02 14:07:46.038062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.031814 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph6xh\" (UniqueName: \"kubernetes.io/projected/809a5f84-4c45-48d1-81b1-926f8e250dba-kube-api-access-ph6xh\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr\" (UID: \"809a5f84-4c45-48d1-81b1-926f8e250dba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:46.038062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.032040 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809a5f84-4c45-48d1-81b1-926f8e250dba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr\" (UID: \"809a5f84-4c45-48d1-81b1-926f8e250dba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:46.038062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.032974 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809a5f84-4c45-48d1-81b1-926f8e250dba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr\" (UID: \"809a5f84-4c45-48d1-81b1-926f8e250dba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:46.150621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.150577 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9bbf\" (UniqueName: \"kubernetes.io/projected/8749ee23-a700-4b18-b691-ecd026e34239-kube-api-access-f9bbf\") pod \"payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv\" (UID: \"8749ee23-a700-4b18-b691-ecd026e34239\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:46.151003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.150959 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8749ee23-a700-4b18-b691-ecd026e34239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv\" (UID: \"8749ee23-a700-4b18-b691-ecd026e34239\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:46.188306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.188208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:46.197577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.197542 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph6xh\" (UniqueName: \"kubernetes.io/projected/809a5f84-4c45-48d1-81b1-926f8e250dba-kube-api-access-ph6xh\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr\" (UID: \"809a5f84-4c45-48d1-81b1-926f8e250dba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:46.256468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.254966 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8749ee23-a700-4b18-b691-ecd026e34239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv\" (UID: \"8749ee23-a700-4b18-b691-ecd026e34239\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:46.256468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.255341 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9bbf\" (UniqueName: \"kubernetes.io/projected/8749ee23-a700-4b18-b691-ecd026e34239-kube-api-access-f9bbf\") pod \"payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv\" (UID: \"8749ee23-a700-4b18-b691-ecd026e34239\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:46.256468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.256427 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8749ee23-a700-4b18-b691-ecd026e34239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv\" (UID: \"8749ee23-a700-4b18-b691-ecd026e34239\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:46.286002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.285943 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9bbf\" (UniqueName: \"kubernetes.io/projected/8749ee23-a700-4b18-b691-ecd026e34239-kube-api-access-f9bbf\") pod \"payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv\" (UID: \"8749ee23-a700-4b18-b691-ecd026e34239\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:46.467625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.467592 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:46.551646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.551596 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:46.777311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.777279 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr"] Nov 02 14:07:46.777854 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:07:46.777834 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809a5f84_4c45_48d1_81b1_926f8e250dba.slice/crio-c4eeb1df5a7fe7739cdff43590e97ffeee229f41791928c6c8531c0a81e36293 WatchSource:0}: Error finding container c4eeb1df5a7fe7739cdff43590e97ffeee229f41791928c6c8531c0a81e36293: Status 404 returned error can't find the container with id c4eeb1df5a7fe7739cdff43590e97ffeee229f41791928c6c8531c0a81e36293 Nov 02 14:07:46.885144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.885116 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv"] Nov 02 14:07:46.887629 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:07:46.887596 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8749ee23_a700_4b18_b691_ecd026e34239.slice/crio-5eb84c0effb0f34802a4bfc940cb6bbc943e1c43e19edda5ffa371db30c41614 WatchSource:0}: Error finding container 5eb84c0effb0f34802a4bfc940cb6bbc943e1c43e19edda5ffa371db30c41614: Status 404 returned error can't find the container with id 5eb84c0effb0f34802a4bfc940cb6bbc943e1c43e19edda5ffa371db30c41614 Nov 02 14:07:47.010037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:46.992914 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:47.227778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.227742 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:47.553269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.553169 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" event={"ID":"809a5f84-4c45-48d1-81b1-926f8e250dba","Type":"ContainerStarted","Data":"42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa"} Nov 02 14:07:47.553269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.553238 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" event={"ID":"809a5f84-4c45-48d1-81b1-926f8e250dba","Type":"ContainerStarted","Data":"c4eeb1df5a7fe7739cdff43590e97ffeee229f41791928c6c8531c0a81e36293"} Nov 02 14:07:47.553269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.553269 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:47.559814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.559761 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" event={"ID":"8749ee23-a700-4b18-b691-ecd026e34239","Type":"ContainerStarted","Data":"c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57"} Nov 02 14:07:47.559926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.559816 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" event={"ID":"8749ee23-a700-4b18-b691-ecd026e34239","Type":"ContainerStarted","Data":"5eb84c0effb0f34802a4bfc940cb6bbc943e1c43e19edda5ffa371db30c41614"} Nov 02 14:07:47.559926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.559847 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:47.601628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.601581 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" podStartSLOduration=2.601569064 podStartE2EDuration="2.601569064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:47.600566756 +0000 UTC m=+1416.326188526" watchObservedRunningTime="2026-11-02 14:07:47.601569064 +0000 UTC m=+1416.327190834" Nov 02 14:07:47.691358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.691306 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" podStartSLOduration=2.69129264 podStartE2EDuration="2.69129264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:47.687199108 +0000 UTC m=+1416.412820877" watchObservedRunningTime="2026-11-02 14:07:47.69129264 +0000 UTC m=+1416.416914409" Nov 02 14:07:47.914116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:47.914044 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:48.223827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:48.223750 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz"] Nov 02 14:07:48.224070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:48.224041 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" containerID="cri-o://5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816" gracePeriod=30 Nov 02 14:07:48.374449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:48.374411 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:49.007827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:49.006690 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:49.175734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:49.175702 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:49.897426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:49.897393 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:51.491051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:51.490968 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:52.912230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:52.912193 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx"] Nov 02 14:07:52.912645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:52.912414 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" podUID="483df060-32e2-4c41-92a9-5be3dc61c54f" containerName="maas-api" containerID="cri-o://8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6" gracePeriod=30 Nov 02 14:07:53.091816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.091765 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:53.132734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.132696 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg"] Nov 02 14:07:53.133227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.132944 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" podUID="6db34dfb-1f13-4549-8dea-2ad2a31e2a04" containerName="payload-processing" containerID="cri-o://7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421" gracePeriod=30 Nov 02 14:07:53.339301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.339232 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:07:53.389846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.389816 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:07:53.408070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.408036 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/483df060-32e2-4c41-92a9-5be3dc61c54f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483df060-32e2-4c41-92a9-5be3dc61c54f-openshift-service-ca\") pod \"483df060-32e2-4c41-92a9-5be3dc61c54f\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " Nov 02 14:07:53.408232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.408111 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls\") pod \"483df060-32e2-4c41-92a9-5be3dc61c54f\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " Nov 02 14:07:53.408232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.408218 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483df060-32e2-4c41-92a9-5be3dc61c54f-kube-api-access-krs8b\" (UniqueName: \"kubernetes.io/projected/483df060-32e2-4c41-92a9-5be3dc61c54f-kube-api-access-krs8b\") pod \"483df060-32e2-4c41-92a9-5be3dc61c54f\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " Nov 02 14:07:53.408373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.408351 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-metrics-certs\") pod \"483df060-32e2-4c41-92a9-5be3dc61c54f\" (UID: \"483df060-32e2-4c41-92a9-5be3dc61c54f\") " Nov 02 14:07:53.408787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.408760 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483df060-32e2-4c41-92a9-5be3dc61c54f-openshift-service-ca" pod "483df060-32e2-4c41-92a9-5be3dc61c54f" (UID: "483df060-32e2-4c41-92a9-5be3dc61c54f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:53.409344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.409319 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483df060-32e2-4c41-92a9-5be3dc61c54f-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.411255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.411222 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:53.411347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.411321 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483df060-32e2-4c41-92a9-5be3dc61c54f-kube-api-access-krs8b" pod "483df060-32e2-4c41-92a9-5be3dc61c54f" (UID: "483df060-32e2-4c41-92a9-5be3dc61c54f"). InnerVolumeSpecName "kube-api-access-krs8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:53.412432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.412403 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls" pod "483df060-32e2-4c41-92a9-5be3dc61c54f" (UID: "483df060-32e2-4c41-92a9-5be3dc61c54f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:07:53.412759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.412737 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-metrics-certs" pod "483df060-32e2-4c41-92a9-5be3dc61c54f" (UID: "483df060-32e2-4c41-92a9-5be3dc61c54f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:07:53.510220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.510179 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-kube-api-access-l49ts\" (UniqueName: \"kubernetes.io/projected/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-kube-api-access-l49ts\") pod \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\" (UID: \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\") " Nov 02 14:07:53.510536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.510518 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-plugins-config-volume\") pod \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\" (UID: \"6db34dfb-1f13-4549-8dea-2ad2a31e2a04\") " Nov 02 14:07:53.511040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.510974 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-plugins-config-volume" pod "6db34dfb-1f13-4549-8dea-2ad2a31e2a04" (UID: "6db34dfb-1f13-4549-8dea-2ad2a31e2a04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:53.511315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.511288 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.511378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.511322 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.511378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.511338 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krs8b\" (UniqueName: \"kubernetes.io/projected/483df060-32e2-4c41-92a9-5be3dc61c54f-kube-api-access-krs8b\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.511378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.511352 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483df060-32e2-4c41-92a9-5be3dc61c54f-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.513031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.512968 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-kube-api-access-l49ts" pod "6db34dfb-1f13-4549-8dea-2ad2a31e2a04" (UID: "6db34dfb-1f13-4549-8dea-2ad2a31e2a04"). InnerVolumeSpecName "kube-api-access-l49ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:53.537603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.537575 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:07:53.632252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.632218 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-token\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-token\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.632583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.632562 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-certs\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.632745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.632727 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-credential-socket\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.632879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.632863 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33351e87-01da-44ee-98c8-e59f6e0aa709-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33351e87-01da-44ee-98c8-e59f6e0aa709-istiod-ca-cert\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.633042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.633026 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-data\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.652747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.652712 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-socket\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.653110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.653089 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-podinfo\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.653483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.653465 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-envoy\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.653688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.653670 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-kube-api-access-djgxg\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-kube-api-access-djgxg\") pod \"33351e87-01da-44ee-98c8-e59f6e0aa709\" (UID: \"33351e87-01da-44ee-98c8-e59f6e0aa709\") " Nov 02 14:07:53.673650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.673610 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l49ts\" (UniqueName: \"kubernetes.io/projected/6db34dfb-1f13-4549-8dea-2ad2a31e2a04-kube-api-access-l49ts\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.692006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.689296 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-credential-socket" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:53.692006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.689463 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-certs" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:53.692006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.689591 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-socket" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:53.692006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.689842 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33351e87-01da-44ee-98c8-e59f6e0aa709-istiod-ca-cert" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:07:53.692006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.689974 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-data" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:53.709393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.709348 5591 generic.go:356] "Generic (PLEG): container finished" podID="6db34dfb-1f13-4549-8dea-2ad2a31e2a04" containerID="7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421" exitCode=0 Nov 02 14:07:53.735459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.709780 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" event={"ID":"6db34dfb-1f13-4549-8dea-2ad2a31e2a04","Type":"ContainerDied","Data":"7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421"} Nov 02 14:07:53.735771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.715435 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" Nov 02 14:07:53.736104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.736060 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg" event={"ID":"6db34dfb-1f13-4549-8dea-2ad2a31e2a04","Type":"ContainerDied","Data":"bf31ba7b33aeb584f2962bbb1141cacdd66f4f984901819cbf30783f6d49fd1f"} Nov 02 14:07:53.736205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.736131 5591 scope.go:122] "RemoveContainer" containerID="7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421" Nov 02 14:07:53.736385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.715644 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:53.769167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.769104 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-kube-api-access-djgxg" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "kube-api-access-djgxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:53.769459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.769426 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-podinfo" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:07:53.769666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.769640 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-token" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:07:53.773823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.773776 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-envoy" pod "33351e87-01da-44ee-98c8-e59f6e0aa709" (UID: "33351e87-01da-44ee-98c8-e59f6e0aa709"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775723 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775750 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgxg\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-kube-api-access-djgxg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775765 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775780 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775791 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775804 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33351e87-01da-44ee-98c8-e59f6e0aa709-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775815 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775825 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33351e87-01da-44ee-98c8-e59f6e0aa709-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.776349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.775836 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33351e87-01da-44ee-98c8-e59f6e0aa709-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:07:53.779081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.778877 5591 generic.go:356] "Generic (PLEG): container finished" podID="483df060-32e2-4c41-92a9-5be3dc61c54f" containerID="8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6" exitCode=0 Nov 02 14:07:53.780099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.779296 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" event={"ID":"483df060-32e2-4c41-92a9-5be3dc61c54f","Type":"ContainerDied","Data":"8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6"} Nov 02 14:07:53.780099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.779356 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" event={"ID":"483df060-32e2-4c41-92a9-5be3dc61c54f","Type":"ContainerDied","Data":"f4e2cdea11de46d3cd5eb03eb2740a047ca142b0ee98a2c2cb61d5ba953264de"} Nov 02 14:07:53.780099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.779472 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx" Nov 02 14:07:53.787524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.787492 5591 generic.go:356] "Generic (PLEG): container finished" podID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerID="5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816" exitCode=0 Nov 02 14:07:53.787764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.787743 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" event={"ID":"33351e87-01da-44ee-98c8-e59f6e0aa709","Type":"ContainerDied","Data":"5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816"} Nov 02 14:07:53.787932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.787916 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" event={"ID":"33351e87-01da-44ee-98c8-e59f6e0aa709","Type":"ContainerDied","Data":"bcc2eafbed78e358c4b817e6d7e1744a4fbd6050c5ffdff22373ac7f518e11a0"} Nov 02 14:07:53.788196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.788179 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz" Nov 02 14:07:53.859921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.859879 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx"] Nov 02 14:07:53.861681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.861657 5591 scope.go:122] "RemoveContainer" containerID="7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421" Nov 02 14:07:53.862128 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:53.862099 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421\": container with ID starting with 7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421 not found: ID does not exist" containerID="7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421" Nov 02 14:07:53.862240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.862143 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421"} err="failed to get container status \"7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421\": rpc error: code = NotFound desc = could not find container \"7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421\": container with ID starting with 7b32da1bcd9c8edad7224dec08301e133f8aea31952c46b41a34f269c22a1421 not found: ID does not exist" Nov 02 14:07:53.862240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.862169 5591 scope.go:122] "RemoveContainer" containerID="8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6" Nov 02 14:07:53.865234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.865208 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ae366a-5c88f9c654-trxcx"] Nov 02 14:07:53.880464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.880440 5591 scope.go:122] "RemoveContainer" containerID="8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6" Nov 02 14:07:53.880850 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:53.880822 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6\": container with ID starting with 8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6 not found: ID does not exist" containerID="8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6" Nov 02 14:07:53.880945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.880864 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6"} err="failed to get container status \"8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6\": rpc error: code = NotFound desc = could not find container \"8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6\": container with ID starting with 8b115f1af8aad606410f5dc67774daa428e28339d26873901d47ab6a3196aff6 not found: ID does not exist" Nov 02 14:07:53.880945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.880890 5591 scope.go:122] "RemoveContainer" containerID="5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816" Nov 02 14:07:53.890880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.890862 5591 scope.go:122] "RemoveContainer" containerID="cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0" Nov 02 14:07:53.900041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.900017 5591 scope.go:122] "RemoveContainer" containerID="5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816" Nov 02 14:07:53.900347 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:53.900327 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816\": container with ID starting with 5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816 not found: ID does not exist" containerID="5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816" Nov 02 14:07:53.900396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.900357 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816"} err="failed to get container status \"5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816\": rpc error: code = NotFound desc = could not find container \"5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816\": container with ID starting with 5a36515449eaf4b16257d7b73fbaa0d6eafdeea2b23453e5014046fea928e816 not found: ID does not exist" Nov 02 14:07:53.900396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.900375 5591 scope.go:122] "RemoveContainer" containerID="cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0" Nov 02 14:07:53.900637 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:07:53.900612 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0\": container with ID starting with cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0 not found: ID does not exist" containerID="cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0" Nov 02 14:07:53.900686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.900648 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0"} err="failed to get container status \"cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0\": rpc error: code = NotFound desc = could not find container \"cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0\": container with ID starting with cd6487d0d2081dbd8389eba5417eb799920b82e1fb1eea727d2ff35a2ea9b4d0 not found: ID does not exist" Nov 02 14:07:53.914066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.914031 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg"] Nov 02 14:07:53.919051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.918976 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ae366a-868c498858-mxqzg"] Nov 02 14:07:53.962558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.962524 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz"] Nov 02 14:07:53.973159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:53.973050 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e543e0-openshift-default-6f884959b7-l58dz"] Nov 02 14:07:54.105699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:54.104135 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:55.433700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.433658 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" path="/var/lib/kubelet/pods/33351e87-01da-44ee-98c8-e59f6e0aa709/volumes" Nov 02 14:07:55.434358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.434338 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="483df060-32e2-4c41-92a9-5be3dc61c54f" path="/var/lib/kubelet/pods/483df060-32e2-4c41-92a9-5be3dc61c54f/volumes" Nov 02 14:07:55.434827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.434808 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db34dfb-1f13-4549-8dea-2ad2a31e2a04" path="/var/lib/kubelet/pods/6db34dfb-1f13-4549-8dea-2ad2a31e2a04/volumes" Nov 02 14:07:55.632522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.632476 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:55.827806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.827702 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:55.996103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.996058 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2"] Nov 02 14:07:55.997282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.997250 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6db34dfb-1f13-4549-8dea-2ad2a31e2a04" containerName="payload-processing" Nov 02 14:07:55.997489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.997469 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db34dfb-1f13-4549-8dea-2ad2a31e2a04" containerName="payload-processing" Nov 02 14:07:55.997666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.997651 5591 cpu_manager.go:414] "containerMap: removing container" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" Nov 02 14:07:55.997750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.997738 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" Nov 02 14:07:55.997855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.997844 5591 cpu_manager.go:414] "containerMap: removing container" podUID="483df060-32e2-4c41-92a9-5be3dc61c54f" containerName="maas-api" Nov 02 14:07:55.997932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.997920 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483df060-32e2-4c41-92a9-5be3dc61c54f" containerName="maas-api" Nov 02 14:07:55.998247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.998230 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" Nov 02 14:07:55.998378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.998350 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db34dfb-1f13-4549-8dea-2ad2a31e2a04" containerName="payload-processing" Nov 02 14:07:55.998518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.998507 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" Nov 02 14:07:55.998600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:55.998590 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="483df060-32e2-4c41-92a9-5be3dc61c54f" containerName="maas-api" Nov 02 14:07:56.024002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.007231 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.053009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.052338 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b7489f\"" Nov 02 14:07:56.073597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.069331 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b7489f-dockercfg-jvcmk\"" Nov 02 14:07:56.080386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.080298 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2"] Nov 02 14:07:56.129235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.127481 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2\" (UID: \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.129235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.127772 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpk8z\" (UniqueName: \"kubernetes.io/projected/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-kube-api-access-mpk8z\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2\" (UID: \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.135621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.135589 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2"] Nov 02 14:07:56.136672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.136652 5591 cpu_manager.go:414] "containerMap: removing container" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" Nov 02 14:07:56.136734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.136676 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33351e87-01da-44ee-98c8-e59f6e0aa709" containerName="istio-proxy" Nov 02 14:07:56.140967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.140947 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:56.161738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.161704 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2"] Nov 02 14:07:56.229022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.228974 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpk8z\" (UniqueName: \"kubernetes.io/projected/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-kube-api-access-mpk8z\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2\" (UID: \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.229212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.229103 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5m7s\" (UniqueName: \"kubernetes.io/projected/4493bf10-287d-48d6-8b88-9cbf447e9496-kube-api-access-m5m7s\") pod \"payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2\" (UID: \"4493bf10-287d-48d6-8b88-9cbf447e9496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:56.229212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.229155 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2\" (UID: \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.229363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.229343 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4493bf10-287d-48d6-8b88-9cbf447e9496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2\" (UID: \"4493bf10-287d-48d6-8b88-9cbf447e9496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:56.229840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.229823 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2\" (UID: \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.256466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.256431 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpk8z\" (UniqueName: \"kubernetes.io/projected/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-kube-api-access-mpk8z\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2\" (UID: \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.331186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.331116 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5m7s\" (UniqueName: \"kubernetes.io/projected/4493bf10-287d-48d6-8b88-9cbf447e9496-kube-api-access-m5m7s\") pod \"payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2\" (UID: \"4493bf10-287d-48d6-8b88-9cbf447e9496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:56.331325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.331221 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4493bf10-287d-48d6-8b88-9cbf447e9496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2\" (UID: \"4493bf10-287d-48d6-8b88-9cbf447e9496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:56.331920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.331898 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4493bf10-287d-48d6-8b88-9cbf447e9496-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2\" (UID: \"4493bf10-287d-48d6-8b88-9cbf447e9496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:56.356242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.356209 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5m7s\" (UniqueName: \"kubernetes.io/projected/4493bf10-287d-48d6-8b88-9cbf447e9496-kube-api-access-m5m7s\") pod \"payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2\" (UID: \"4493bf10-287d-48d6-8b88-9cbf447e9496\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:56.381043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.381017 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.461042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.458397 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:56.698008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.697964 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2"] Nov 02 14:07:56.699260 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:07:56.699233 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69dd95b_ae85_40f1_8b41_a5c29a2a89ac.slice/crio-36c1ca517aab54bbd64367fedf9a52c56d64fa58eeb34908d4228111ca34500d WatchSource:0}: Error finding container 36c1ca517aab54bbd64367fedf9a52c56d64fa58eeb34908d4228111ca34500d: Status 404 returned error can't find the container with id 36c1ca517aab54bbd64367fedf9a52c56d64fa58eeb34908d4228111ca34500d Nov 02 14:07:56.737790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.737764 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:56.813753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.813720 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" event={"ID":"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac","Type":"ContainerStarted","Data":"53192e5c0494562ecc83dbd3537a96c04d851af2570051399365c6bac1df0f4f"} Nov 02 14:07:56.813944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.813773 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" event={"ID":"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac","Type":"ContainerStarted","Data":"36c1ca517aab54bbd64367fedf9a52c56d64fa58eeb34908d4228111ca34500d"} Nov 02 14:07:56.813944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.813815 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:07:56.852605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.852580 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2"] Nov 02 14:07:56.853999 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:07:56.853964 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4493bf10_287d_48d6_8b88_9cbf447e9496.slice/crio-0a28ac03609eb4f7cf5ffc3b189eaf910a1044cb2e4d180c0d3c2f63ed2b131c WatchSource:0}: Error finding container 0a28ac03609eb4f7cf5ffc3b189eaf910a1044cb2e4d180c0d3c2f63ed2b131c: Status 404 returned error can't find the container with id 0a28ac03609eb4f7cf5ffc3b189eaf910a1044cb2e4d180c0d3c2f63ed2b131c Nov 02 14:07:56.907022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:56.898393 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" podStartSLOduration=1.898370122 podStartE2EDuration="1.898370122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:56.887504739 +0000 UTC m=+1425.613126510" watchObservedRunningTime="2026-11-02 14:07:56.898370122 +0000 UTC m=+1425.623991928" Nov 02 14:07:57.044023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:57.039884 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:57.825419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:57.825370 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" event={"ID":"4493bf10-287d-48d6-8b88-9cbf447e9496","Type":"ContainerStarted","Data":"134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8"} Nov 02 14:07:57.825847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:57.825472 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" event={"ID":"4493bf10-287d-48d6-8b88-9cbf447e9496","Type":"ContainerStarted","Data":"0a28ac03609eb4f7cf5ffc3b189eaf910a1044cb2e4d180c0d3c2f63ed2b131c"} Nov 02 14:07:57.825847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:57.825708 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:07:57.876267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:57.876216 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" podStartSLOduration=1.87620185 podStartE2EDuration="1.87620185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:57.872723832 +0000 UTC m=+1426.598345601" watchObservedRunningTime="2026-11-02 14:07:57.87620185 +0000 UTC m=+1426.601823620" Nov 02 14:07:58.597393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:58.595059 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:07:58.597393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:58.595696 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:07:58.688952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:58.688923 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:58.930688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:58.930652 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:59.364317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:59.364232 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:07:59.711289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:59.711258 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh"] Nov 02 14:07:59.711496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:59.711457 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" containerID="cri-o://f75ca4488842778333bebf3c286dcc22202f74e6c9c0b6e6c27b1fa77d0cb0d2" gracePeriod=30 Nov 02 14:07:59.867012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:07:59.865381 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:00.719796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:00.719765 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:01.074800 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:01.074718 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:08:01.522479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:01.522442 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:03.184302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:03.184262 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:08:03.933190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:03.930892 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:04.929499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:04.929467 5591 generic.go:356] "Generic (PLEG): container finished" podID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerID="f75ca4488842778333bebf3c286dcc22202f74e6c9c0b6e6c27b1fa77d0cb0d2" exitCode=0 Nov 02 14:08:04.929910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:04.929551 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" event={"ID":"df2486b6-a99e-4bee-939e-88263acdd4b7","Type":"ContainerDied","Data":"f75ca4488842778333bebf3c286dcc22202f74e6c9c0b6e6c27b1fa77d0cb0d2"} Nov 02 14:08:04.929910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:04.929630 5591 scope.go:122] "RemoveContainer" containerID="215d88483293379475e25fcecf915b2f1d9b1d9efee9b66cde1cd71f1a46db35" Nov 02 14:08:05.004145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.004117 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:08:05.101549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101512 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-kube-api-access-frxpj\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-kube-api-access-frxpj\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.101720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101614 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-data\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.101720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101657 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-credential-socket\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.101720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101710 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-token\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-token\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.101897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101741 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-certs\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.101897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101823 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2486b6-a99e-4bee-939e-88263acdd4b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2486b6-a99e-4bee-939e-88263acdd4b7-istiod-ca-cert\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.102030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101937 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-podinfo\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.102030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101941 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-data" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:05.102030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.101998 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-credential-socket" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:05.102212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.102042 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-socket\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.102212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.102075 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-envoy\") pod \"df2486b6-a99e-4bee-939e-88263acdd4b7\" (UID: \"df2486b6-a99e-4bee-939e-88263acdd4b7\") " Nov 02 14:08:05.102212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.102164 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-certs" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:05.102363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.102295 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2486b6-a99e-4bee-939e-88263acdd4b7-istiod-ca-cert" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:05.102723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.102696 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.102811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.102730 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.102811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.102746 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.102811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.102761 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2486b6-a99e-4bee-939e-88263acdd4b7-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.103091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.103066 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-socket" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:05.104525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.104496 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-kube-api-access-frxpj" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "kube-api-access-frxpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:05.104852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.104826 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-podinfo" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:08:05.104966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.104937 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-token" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:05.105113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.105093 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-envoy" pod "df2486b6-a99e-4bee-939e-88263acdd4b7" (UID: "df2486b6-a99e-4bee-939e-88263acdd4b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:05.204326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.204289 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.204326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.204321 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.204326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.204331 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.204570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.204341 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2486b6-a99e-4bee-939e-88263acdd4b7-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.204570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.204351 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxpj\" (UniqueName: \"kubernetes.io/projected/df2486b6-a99e-4bee-939e-88263acdd4b7-kube-api-access-frxpj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:05.367444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.367344 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:05.977440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.973198 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" event={"ID":"df2486b6-a99e-4bee-939e-88263acdd4b7","Type":"ContainerDied","Data":"4c8f229f6c33aa4caa37013eb4dff0e265cec49c922ff9cfe5541ec30924bae4"} Nov 02 14:08:05.977440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.973274 5591 scope.go:122] "RemoveContainer" containerID="f75ca4488842778333bebf3c286dcc22202f74e6c9c0b6e6c27b1fa77d0cb0d2" Nov 02 14:08:05.977440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:05.973390 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh" Nov 02 14:08:06.051724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:06.051686 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh"] Nov 02 14:08:06.055446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:06.055419 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ae366a-openshift-default-59fd8dbbd7-g2bqh"] Nov 02 14:08:06.178006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:06.177074 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:07.434968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:07.434931 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" path="/var/lib/kubelet/pods/df2486b6-a99e-4bee-939e-88263acdd4b7/volumes" Nov 02 14:08:07.828196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:07.828113 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:08:07.864164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:07.864130 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:07.939868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:07.939837 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww"] Nov 02 14:08:07.940149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:07.940115 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" podUID="ea44bf2f-bc55-4b19-8e44-42f3072271c8" containerName="payload-pre-processing" containerID="cri-o://3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02" gracePeriod=30 Nov 02 14:08:08.062256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.062174 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:08.225320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.225287 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:08:08.230499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.230472 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:08.304002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.303661 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea44bf2f-bc55-4b19-8e44-42f3072271c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea44bf2f-bc55-4b19-8e44-42f3072271c8-plugins-config-volume\") pod \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\" (UID: \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\") " Nov 02 14:08:08.304002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.303794 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea44bf2f-bc55-4b19-8e44-42f3072271c8-kube-api-access-2fmh8\" (UniqueName: \"kubernetes.io/projected/ea44bf2f-bc55-4b19-8e44-42f3072271c8-kube-api-access-2fmh8\") pod \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\" (UID: \"ea44bf2f-bc55-4b19-8e44-42f3072271c8\") " Nov 02 14:08:08.350162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.350120 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea44bf2f-bc55-4b19-8e44-42f3072271c8-plugins-config-volume" pod "ea44bf2f-bc55-4b19-8e44-42f3072271c8" (UID: "ea44bf2f-bc55-4b19-8e44-42f3072271c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:08.494546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.494515 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea44bf2f-bc55-4b19-8e44-42f3072271c8-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:08.551002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.550187 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea44bf2f-bc55-4b19-8e44-42f3072271c8-kube-api-access-2fmh8" pod "ea44bf2f-bc55-4b19-8e44-42f3072271c8" (UID: "ea44bf2f-bc55-4b19-8e44-42f3072271c8"). InnerVolumeSpecName "kube-api-access-2fmh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:08.621653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.620780 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fmh8\" (UniqueName: \"kubernetes.io/projected/ea44bf2f-bc55-4b19-8e44-42f3072271c8-kube-api-access-2fmh8\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:08.622924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.622895 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh"] Nov 02 14:08:08.624068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.624042 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ea44bf2f-bc55-4b19-8e44-42f3072271c8" containerName="payload-pre-processing" Nov 02 14:08:08.624246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.624228 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea44bf2f-bc55-4b19-8e44-42f3072271c8" containerName="payload-pre-processing" Nov 02 14:08:08.624411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.624390 5591 cpu_manager.go:414] "containerMap: removing container" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" Nov 02 14:08:08.624522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.624505 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" Nov 02 14:08:08.624630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.624620 5591 cpu_manager.go:414] "containerMap: removing container" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" Nov 02 14:08:08.624704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.624693 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" Nov 02 14:08:08.625025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.625010 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" Nov 02 14:08:08.625121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.625112 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2486b6-a99e-4bee-939e-88263acdd4b7" containerName="istio-proxy" Nov 02 14:08:08.625219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.625207 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea44bf2f-bc55-4b19-8e44-42f3072271c8" containerName="payload-pre-processing" Nov 02 14:08:08.647757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.647731 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.655003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.654952 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh"] Nov 02 14:08:08.667701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.667182 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-54f2fe-serving-cert\"" Nov 02 14:08:08.667701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.667501 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:08.721814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.721721 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.721814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.721762 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-metrics-certs\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.722088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.721852 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmpfw\" (UniqueName: \"kubernetes.io/projected/816d3fdb-8e14-4eda-b40d-3a4da1499c91-kube-api-access-hmpfw\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.722088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.721909 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d3fdb-8e14-4eda-b40d-3a4da1499c91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.835006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.834370 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.835006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.834439 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-metrics-certs\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.835006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.834550 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmpfw\" (UniqueName: \"kubernetes.io/projected/816d3fdb-8e14-4eda-b40d-3a4da1499c91-kube-api-access-hmpfw\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.835006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.834628 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d3fdb-8e14-4eda-b40d-3a4da1499c91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.835707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.835681 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:08:08.873011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.849035 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.873011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.850345 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:08.873011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.871782 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d3fdb-8e14-4eda-b40d-3a4da1499c91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.882010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.876580 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-metrics-certs\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.883396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.882687 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmpfw\" (UniqueName: \"kubernetes.io/projected/816d3fdb-8e14-4eda-b40d-3a4da1499c91-kube-api-access-hmpfw\") pod \"maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.977121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.977033 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:08.997718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.997688 5591 generic.go:356] "Generic (PLEG): container finished" podID="ea44bf2f-bc55-4b19-8e44-42f3072271c8" containerID="3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02" exitCode=0 Nov 02 14:08:08.997856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.997774 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" Nov 02 14:08:08.997856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.997776 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" event={"ID":"ea44bf2f-bc55-4b19-8e44-42f3072271c8","Type":"ContainerDied","Data":"3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02"} Nov 02 14:08:08.997943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.997887 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww" event={"ID":"ea44bf2f-bc55-4b19-8e44-42f3072271c8","Type":"ContainerDied","Data":"a85a346ba488349c692c1b5ddb72dd1aad7d2e754feffc87ede29deca423c7cd"} Nov 02 14:08:08.997943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:08.997921 5591 scope.go:122] "RemoveContainer" containerID="3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02" Nov 02 14:08:09.016843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.016816 5591 scope.go:122] "RemoveContainer" containerID="3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02" Nov 02 14:08:09.017191 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:09.017169 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02\": container with ID starting with 3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02 not found: ID does not exist" containerID="3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02" Nov 02 14:08:09.017249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.017200 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02"} err="failed to get container status \"3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02\": rpc error: code = NotFound desc = could not find container \"3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02\": container with ID starting with 3192657846afe122c7d2851d2eedb0c2385a10b3f3bd308cbc2861db293ebe02 not found: ID does not exist" Nov 02 14:08:09.066952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.066920 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww"] Nov 02 14:08:09.073401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.073374 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fb206f-5948cbc95-84lww"] Nov 02 14:08:09.326324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.326216 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh"] Nov 02 14:08:09.374215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.374104 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:09.433647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.433615 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea44bf2f-bc55-4b19-8e44-42f3072271c8" path="/var/lib/kubelet/pods/ea44bf2f-bc55-4b19-8e44-42f3072271c8/volumes" Nov 02 14:08:09.679248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.679213 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f"] Nov 02 14:08:09.679688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.679492 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" containerID="cri-o://a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a" gracePeriod=30 Nov 02 14:08:09.774769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:09.774696 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:10.090184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:10.090137 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" event={"ID":"816d3fdb-8e14-4eda-b40d-3a4da1499c91","Type":"ContainerStarted","Data":"fe3645a7549a8b0f797de52105b74e338337ee99fbd28673cd8a4b3b17ab5ca0"} Nov 02 14:08:10.766018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:10.762310 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:10.970189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:10.970154 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:11.100150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.100052 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" event={"ID":"816d3fdb-8e14-4eda-b40d-3a4da1499c91","Type":"ContainerStarted","Data":"21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671"} Nov 02 14:08:11.100150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.100120 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:11.146367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.146331 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98"] Nov 02 14:08:11.151409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.151373 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.157778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.157719 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" podStartSLOduration=2.747420109 podStartE2EDuration="3.157699155s" totalImagesPullingTime="410.279046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:11.152220471 +0000 UTC m=+1439.877842242" watchObservedRunningTime="2026-11-02 14:08:11.157699155 +0000 UTC m=+1439.883320925" Nov 02 14:08:11.178692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.178667 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-13367a-openshift-default-dockercfg-58t82\"" Nov 02 14:08:11.194969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.194930 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-credential-socket\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.195211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.195006 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-podinfo\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.195211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.195116 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-certs\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.195211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.195147 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istiod-ca-cert\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.195211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.195174 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-data\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.195449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.195350 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-token\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.195449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.195407 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlb2n\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-kube-api-access-vlb2n\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.195559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.195477 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-envoy\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.195559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.195521 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-socket\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.198670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.198642 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98"] Nov 02 14:08:11.271956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.271908 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:11.300514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.300471 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-credential-socket\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.301469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.301311 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-credential-socket\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.308052 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-podinfo\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.308190 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-certs\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.308226 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istiod-ca-cert\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.308258 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-data\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.308400 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-token\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.308458 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlb2n\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-kube-api-access-vlb2n\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.308519 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-envoy\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.308561 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-socket\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.309094 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-socket\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.309341 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-certs\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.310095 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istiod-ca-cert\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.332936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.310372 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-data\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.428969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.428927 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-envoy\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.439095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.439064 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlb2n\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-kube-api-access-vlb2n\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.510033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.490306 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-podinfo\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.510033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.490940 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:11.510033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.492490 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-token\") pod \"e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:11.788222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.788142 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-13367a-openshift-default-dockercfg-58t82\"" Nov 02 14:08:11.791064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:11.791037 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:12.123146 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:08:12.123116 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc868206f_cc9a_4ba2_bcd0_f4f2fb8e03bc.slice/crio-d91825da94fd21ae328542a55ec1a57552d1d96fc6a22a331fafee82d0d67207 WatchSource:0}: Error finding container d91825da94fd21ae328542a55ec1a57552d1d96fc6a22a331fafee82d0d67207: Status 404 returned error can't find the container with id d91825da94fd21ae328542a55ec1a57552d1d96fc6a22a331fafee82d0d67207 Nov 02 14:08:12.123227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:12.123161 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98"] Nov 02 14:08:12.126578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:12.126545 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:12.126643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:12.126628 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:12.126687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:12.126673 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:12.425086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:12.425040 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:12.768435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:12.768349 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:13.094434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:13.094347 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:13.114638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:13.114603 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" event={"ID":"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc","Type":"ContainerStarted","Data":"362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199"} Nov 02 14:08:13.114786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:13.114661 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" event={"ID":"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc","Type":"ContainerStarted","Data":"d91825da94fd21ae328542a55ec1a57552d1d96fc6a22a331fafee82d0d67207"} Nov 02 14:08:13.167504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:13.167446 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" podStartSLOduration=2.167431247 podStartE2EDuration="2.167431247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:13.166784808 +0000 UTC m=+1441.892406580" watchObservedRunningTime="2026-11-02 14:08:13.167431247 +0000 UTC m=+1441.893053013" Nov 02 14:08:13.321011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:13.312225 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:13.791375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:13.791342 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:13.797439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:13.797412 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:14.010719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:14.010688 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:14.120424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:14.120342 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:14.121482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:14.121460 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:08:14.443022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:14.438276 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:14.853787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:14.853607 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:15.018530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.018507 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:08:15.086950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.086913 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55839559-f783-40f1-b436-3e4af057ee6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55839559-f783-40f1-b436-3e4af057ee6a-istio-podinfo\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.087163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087041 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-kube-api-access-484t2\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-kube-api-access-484t2\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.087163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087098 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-envoy\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.087163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087135 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-data\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.087342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087165 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-istio-token\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-istio-token\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.087342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087203 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-socket\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.087342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087240 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-certs\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.087342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087319 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-credential-socket\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.087554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087355 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55839559-f783-40f1-b436-3e4af057ee6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55839559-f783-40f1-b436-3e4af057ee6a-istiod-ca-cert\") pod \"55839559-f783-40f1-b436-3e4af057ee6a\" (UID: \"55839559-f783-40f1-b436-3e4af057ee6a\") " Nov 02 14:08:15.088939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.087602 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-socket" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:15.088939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.088184 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-credential-socket" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:15.088939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.088200 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.088939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.088796 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55839559-f783-40f1-b436-3e4af057ee6a-istiod-ca-cert" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:15.088939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.088864 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-certs" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:15.088939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.088915 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-data" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:15.089756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.089716 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-envoy" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:15.089862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.089802 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-kube-api-access-484t2" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "kube-api-access-484t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:15.091816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.091789 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55839559-f783-40f1-b436-3e4af057ee6a-istio-podinfo" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:08:15.108513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.108487 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-istio-token" pod "55839559-f783-40f1-b436-3e4af057ee6a" (UID: "55839559-f783-40f1-b436-3e4af057ee6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:15.128822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.128792 5591 generic.go:356] "Generic (PLEG): container finished" podID="55839559-f783-40f1-b436-3e4af057ee6a" containerID="a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a" exitCode=0 Nov 02 14:08:15.129255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.128844 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" Nov 02 14:08:15.129255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.128874 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" event={"ID":"55839559-f783-40f1-b436-3e4af057ee6a","Type":"ContainerDied","Data":"a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a"} Nov 02 14:08:15.129255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.128945 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f" event={"ID":"55839559-f783-40f1-b436-3e4af057ee6a","Type":"ContainerDied","Data":"039966c0df921bc44dcd3b450a9eb87212786a2edf4bbacdb0a73d1181e4ae1a"} Nov 02 14:08:15.129255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.128964 5591 scope.go:122] "RemoveContainer" containerID="a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a" Nov 02 14:08:15.139708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.139686 5591 scope.go:122] "RemoveContainer" containerID="a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a" Nov 02 14:08:15.140083 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:15.140035 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a\": container with ID starting with a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a not found: ID does not exist" containerID="a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a" Nov 02 14:08:15.140192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.140081 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a"} err="failed to get container status \"a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a\": rpc error: code = NotFound desc = could not find container \"a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a\": container with ID starting with a8f44857d165ef692f3f6d2435add8b1c1e7a8d8153d106fc501efbff410904a not found: ID does not exist" Nov 02 14:08:15.189638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.189605 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55839559-f783-40f1-b436-3e4af057ee6a-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.189638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.189638 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-484t2\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-kube-api-access-484t2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.189842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.189653 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.189842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.189667 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.189842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.189680 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55839559-f783-40f1-b436-3e4af057ee6a-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.189842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.189691 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.189842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.189702 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55839559-f783-40f1-b436-3e4af057ee6a-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.189842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.189712 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55839559-f783-40f1-b436-3e4af057ee6a-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:15.196164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.196141 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f"] Nov 02 14:08:15.243561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.243486 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-tht2f"] Nov 02 14:08:15.432379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.432341 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55839559-f783-40f1-b436-3e4af057ee6a" path="/var/lib/kubelet/pods/55839559-f783-40f1-b436-3e4af057ee6a/volumes" Nov 02 14:08:15.793819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.793780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:15.969578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:15.969548 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:16.164608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:16.164576 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:16.495654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:16.495580 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:16.697642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:16.696106 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:17.113647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.113615 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:08:17.564094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.564054 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:17.654494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.654460 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht"] Nov 02 14:08:17.655132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.655117 5591 cpu_manager.go:414] "containerMap: removing container" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" Nov 02 14:08:17.655195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.655134 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" Nov 02 14:08:17.655285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.655276 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="55839559-f783-40f1-b436-3e4af057ee6a" containerName="istio-proxy" Nov 02 14:08:17.660170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.660153 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:17.677345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.677308 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-13367a-dockercfg-f57jx\"" Nov 02 14:08:17.677345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.677311 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-13367a\"" Nov 02 14:08:17.678809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.678784 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht"] Nov 02 14:08:17.746008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.743098 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht\" (UID: \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:17.746008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.743283 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-688md\" (UniqueName: \"kubernetes.io/projected/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-kube-api-access-688md\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht\" (UID: \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:17.802008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.799778 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:17.858067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.856192 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-688md\" (UniqueName: \"kubernetes.io/projected/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-kube-api-access-688md\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht\" (UID: \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:17.858067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.856476 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht\" (UID: \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:17.858067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.857457 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht\" (UID: \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:17.979872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.978382 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-688md\" (UniqueName: \"kubernetes.io/projected/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-kube-api-access-688md\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht\" (UID: \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:17.981238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:17.981127 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:18.356554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:18.356529 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht"] Nov 02 14:08:18.358287 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:08:18.358256 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab5b908_0b0c_4f49_83c4_2a8204293fe1.slice/crio-123a067b249f04d71fde67facb68c2b684f539db171c490d4e2bbbb1069ccc59 WatchSource:0}: Error finding container 123a067b249f04d71fde67facb68c2b684f539db171c490d4e2bbbb1069ccc59: Status 404 returned error can't find the container with id 123a067b249f04d71fde67facb68c2b684f539db171c490d4e2bbbb1069ccc59 Nov 02 14:08:19.193521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:19.193485 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" event={"ID":"6ab5b908-0b0c-4f49-83c4-2a8204293fe1","Type":"ContainerStarted","Data":"45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9"} Nov 02 14:08:19.194198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:19.193537 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" event={"ID":"6ab5b908-0b0c-4f49-83c4-2a8204293fe1","Type":"ContainerStarted","Data":"123a067b249f04d71fde67facb68c2b684f539db171c490d4e2bbbb1069ccc59"} Nov 02 14:08:19.194198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:19.193632 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:19.250407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:19.250352 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" podStartSLOduration=2.250338752 podStartE2EDuration="2.250338752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:19.247830021 +0000 UTC m=+1447.973451792" watchObservedRunningTime="2026-11-02 14:08:19.250338752 +0000 UTC m=+1447.975960521" Nov 02 14:08:20.036775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:20.036737 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:21.633174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:21.633142 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:22.238940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:22.238901 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:22.427335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:22.427280 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:22.665052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:22.648896 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:22.840294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:22.840209 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:23.921262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:23.921225 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m"] Nov 02 14:08:23.921775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:23.921465 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" podUID="15fd5322-ed09-4fb3-87b1-e27d578a7af6" containerName="maas-api" containerID="cri-o://582539ea4babe74343ee51a8e657437a351133af405ab1a9ab0bd956cbc9f7cf" gracePeriod=30 Nov 02 14:08:24.098348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.098124 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:24.168102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.168055 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82"] Nov 02 14:08:24.168738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.168349 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" podUID="efa0bbfa-7b4f-46b6-8850-7e425d9d2a66" containerName="payload-pre-processing" containerID="cri-o://448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829" gracePeriod=30 Nov 02 14:08:24.225916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.225888 5591 generic.go:356] "Generic (PLEG): container finished" podID="15fd5322-ed09-4fb3-87b1-e27d578a7af6" containerID="582539ea4babe74343ee51a8e657437a351133af405ab1a9ab0bd956cbc9f7cf" exitCode=0 Nov 02 14:08:24.226089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.225958 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" event={"ID":"15fd5322-ed09-4fb3-87b1-e27d578a7af6","Type":"ContainerDied","Data":"582539ea4babe74343ee51a8e657437a351133af405ab1a9ab0bd956cbc9f7cf"} Nov 02 14:08:24.226089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.226052 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" event={"ID":"15fd5322-ed09-4fb3-87b1-e27d578a7af6","Type":"ContainerDied","Data":"bc6200b42933c380fa006297ada21f3a3c7681153f1ce7176bc8a82eb2e1831c"} Nov 02 14:08:24.226089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.226084 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc6200b42933c380fa006297ada21f3a3c7681153f1ce7176bc8a82eb2e1831c" Nov 02 14:08:24.272599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.272553 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:24.277425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.277398 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:08:24.325649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.325200 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fd5322-ed09-4fb3-87b1-e27d578a7af6-kube-api-access-nfj8z\" (UniqueName: \"kubernetes.io/projected/15fd5322-ed09-4fb3-87b1-e27d578a7af6-kube-api-access-nfj8z\") pod \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " Nov 02 14:08:24.325649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.325318 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15fd5322-ed09-4fb3-87b1-e27d578a7af6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fd5322-ed09-4fb3-87b1-e27d578a7af6-openshift-service-ca\") pod \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " Nov 02 14:08:24.325649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.325455 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-metrics-certs\") pod \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " Nov 02 14:08:24.325649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.325488 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-maas-api-tls\") pod \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\" (UID: \"15fd5322-ed09-4fb3-87b1-e27d578a7af6\") " Nov 02 14:08:24.326017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.325787 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fd5322-ed09-4fb3-87b1-e27d578a7af6-openshift-service-ca" pod "15fd5322-ed09-4fb3-87b1-e27d578a7af6" (UID: "15fd5322-ed09-4fb3-87b1-e27d578a7af6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:24.326200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.326179 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15fd5322-ed09-4fb3-87b1-e27d578a7af6-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:24.327736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.327707 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fd5322-ed09-4fb3-87b1-e27d578a7af6-kube-api-access-nfj8z" pod "15fd5322-ed09-4fb3-87b1-e27d578a7af6" (UID: "15fd5322-ed09-4fb3-87b1-e27d578a7af6"). InnerVolumeSpecName "kube-api-access-nfj8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:24.327895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.327865 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-metrics-certs" pod "15fd5322-ed09-4fb3-87b1-e27d578a7af6" (UID: "15fd5322-ed09-4fb3-87b1-e27d578a7af6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:08:24.328118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.328095 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-maas-api-tls" pod "15fd5322-ed09-4fb3-87b1-e27d578a7af6" (UID: "15fd5322-ed09-4fb3-87b1-e27d578a7af6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:08:24.428415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.428331 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfj8z\" (UniqueName: \"kubernetes.io/projected/15fd5322-ed09-4fb3-87b1-e27d578a7af6-kube-api-access-nfj8z\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:24.428415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.428361 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:24.428415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.428374 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15fd5322-ed09-4fb3-87b1-e27d578a7af6-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:24.441971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.434052 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:24.468675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.468641 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:08:24.529376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.529343 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-kube-api-access-xfm8m\" (UniqueName: \"kubernetes.io/projected/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-kube-api-access-xfm8m\") pod \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\" (UID: \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\") " Nov 02 14:08:24.529550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.529396 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-plugins-config-volume\") pod \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\" (UID: \"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66\") " Nov 02 14:08:24.531165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.531139 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-plugins-config-volume" pod "efa0bbfa-7b4f-46b6-8850-7e425d9d2a66" (UID: "efa0bbfa-7b4f-46b6-8850-7e425d9d2a66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:24.533104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.533064 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-kube-api-access-xfm8m" pod "efa0bbfa-7b4f-46b6-8850-7e425d9d2a66" (UID: "efa0bbfa-7b4f-46b6-8850-7e425d9d2a66"). InnerVolumeSpecName "kube-api-access-xfm8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:24.604691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.604654 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:24.616896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.616861 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x"] Nov 02 14:08:24.617137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.617114 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" podUID="cf6caca4-7449-4a83-b2b1-7747ff0e2fc9" containerName="payload-processing" containerID="cri-o://8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f" gracePeriod=30 Nov 02 14:08:24.631531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.631504 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfm8m\" (UniqueName: \"kubernetes.io/projected/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-kube-api-access-xfm8m\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:24.631531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.631529 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:24.640634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.640601 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz"] Nov 02 14:08:24.640905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.640834 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" podUID="4fce9f56-4586-479f-b01b-7adae0c869f4" containerName="payload-pre-processing" containerID="cri-o://1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8" gracePeriod=30 Nov 02 14:08:24.805399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.805358 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:24.907423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.907380 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:08:24.917224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:24.917198 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:08:25.179034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.175442 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:25.257540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.257504 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fce9f56-4586-479f-b01b-7adae0c869f4-kube-api-access-q75rs\" (UniqueName: \"kubernetes.io/projected/4fce9f56-4586-479f-b01b-7adae0c869f4-kube-api-access-q75rs\") pod \"4fce9f56-4586-479f-b01b-7adae0c869f4\" (UID: \"4fce9f56-4586-479f-b01b-7adae0c869f4\") " Nov 02 14:08:25.258480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.258456 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-kube-api-access-n8svp\" (UniqueName: \"kubernetes.io/projected/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-kube-api-access-n8svp\") pod \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\" (UID: \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\") " Nov 02 14:08:25.258665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.258653 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fce9f56-4586-479f-b01b-7adae0c869f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fce9f56-4586-479f-b01b-7adae0c869f4-plugins-config-volume\") pod \"4fce9f56-4586-479f-b01b-7adae0c869f4\" (UID: \"4fce9f56-4586-479f-b01b-7adae0c869f4\") " Nov 02 14:08:25.259367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.259132 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-plugins-config-volume\") pod \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\" (UID: \"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9\") " Nov 02 14:08:25.259677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.259656 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fce9f56-4586-479f-b01b-7adae0c869f4-plugins-config-volume" pod "4fce9f56-4586-479f-b01b-7adae0c869f4" (UID: "4fce9f56-4586-479f-b01b-7adae0c869f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:25.259784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.259728 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-plugins-config-volume" pod "cf6caca4-7449-4a83-b2b1-7747ff0e2fc9" (UID: "cf6caca4-7449-4a83-b2b1-7747ff0e2fc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:25.260368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.260342 5591 generic.go:356] "Generic (PLEG): container finished" podID="efa0bbfa-7b4f-46b6-8850-7e425d9d2a66" containerID="448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829" exitCode=0 Nov 02 14:08:25.260487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.260473 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" Nov 02 14:08:25.260960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.260937 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" event={"ID":"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66","Type":"ContainerDied","Data":"448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829"} Nov 02 14:08:25.261440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.261412 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82" event={"ID":"efa0bbfa-7b4f-46b6-8850-7e425d9d2a66","Type":"ContainerDied","Data":"d73c7aa96db3c0b8a9ce533a7b6552a1d94895fabaf6ebdaf8cd609ad90ec62d"} Nov 02 14:08:25.261601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.261586 5591 scope.go:122] "RemoveContainer" containerID="448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829" Nov 02 14:08:25.262172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.262146 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-kube-api-access-n8svp" pod "cf6caca4-7449-4a83-b2b1-7747ff0e2fc9" (UID: "cf6caca4-7449-4a83-b2b1-7747ff0e2fc9"). InnerVolumeSpecName "kube-api-access-n8svp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:25.262268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.262188 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fce9f56-4586-479f-b01b-7adae0c869f4-kube-api-access-q75rs" pod "4fce9f56-4586-479f-b01b-7adae0c869f4" (UID: "4fce9f56-4586-479f-b01b-7adae0c869f4"). InnerVolumeSpecName "kube-api-access-q75rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:25.264613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.264593 5591 generic.go:356] "Generic (PLEG): container finished" podID="cf6caca4-7449-4a83-b2b1-7747ff0e2fc9" containerID="8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f" exitCode=0 Nov 02 14:08:25.264708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.264626 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" event={"ID":"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9","Type":"ContainerDied","Data":"8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f"} Nov 02 14:08:25.264708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.264695 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" event={"ID":"cf6caca4-7449-4a83-b2b1-7747ff0e2fc9","Type":"ContainerDied","Data":"4c97db9d277be2b9d6740ad17c3433032313b217ae37fa5c5148eb9233356348"} Nov 02 14:08:25.268035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.266344 5591 generic.go:356] "Generic (PLEG): container finished" podID="4fce9f56-4586-479f-b01b-7adae0c869f4" containerID="1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8" exitCode=0 Nov 02 14:08:25.268035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.266388 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" Nov 02 14:08:25.268035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.266490 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" event={"ID":"4fce9f56-4586-479f-b01b-7adae0c869f4","Type":"ContainerDied","Data":"1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8"} Nov 02 14:08:25.268035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.266529 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz" event={"ID":"4fce9f56-4586-479f-b01b-7adae0c869f4","Type":"ContainerDied","Data":"7ce890dece67431765f880604a7fb10cc32aed5e987011006d04e59839bda430"} Nov 02 14:08:25.268035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.266689 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m" Nov 02 14:08:25.273749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.264968 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x" Nov 02 14:08:25.292193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.292174 5591 scope.go:122] "RemoveContainer" containerID="448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829" Nov 02 14:08:25.292554 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:25.292531 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829\": container with ID starting with 448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829 not found: ID does not exist" containerID="448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829" Nov 02 14:08:25.292687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.292558 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829"} err="failed to get container status \"448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829\": rpc error: code = NotFound desc = could not find container \"448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829\": container with ID starting with 448c16338e798de277c8e301827085a47d4e9ecd7e5db2c4b05833c3e30f5829 not found: ID does not exist" Nov 02 14:08:25.292687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.292575 5591 scope.go:122] "RemoveContainer" containerID="8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f" Nov 02 14:08:25.301747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.301721 5591 scope.go:122] "RemoveContainer" containerID="8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f" Nov 02 14:08:25.301978 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:25.301960 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f\": container with ID starting with 8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f not found: ID does not exist" containerID="8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f" Nov 02 14:08:25.302045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.302011 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f"} err="failed to get container status \"8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f\": rpc error: code = NotFound desc = could not find container \"8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f\": container with ID starting with 8b253ff470cfb00a078f4d814deda87eea3f5d62910d92cd4f6d544bad0ffb1f not found: ID does not exist" Nov 02 14:08:25.302045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.302028 5591 scope.go:122] "RemoveContainer" containerID="1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8" Nov 02 14:08:25.310395 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.310379 5591 scope.go:122] "RemoveContainer" containerID="1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8" Nov 02 14:08:25.310632 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:25.310614 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8\": container with ID starting with 1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8 not found: ID does not exist" containerID="1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8" Nov 02 14:08:25.310690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.310639 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8"} err="failed to get container status \"1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8\": rpc error: code = NotFound desc = could not find container \"1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8\": container with ID starting with 1c06827ecd19450d6357b8aaef4e803d10cc932008b46a295bf3febf81c3e0c8 not found: ID does not exist" Nov 02 14:08:25.361237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.361206 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:25.361237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.361236 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75rs\" (UniqueName: \"kubernetes.io/projected/4fce9f56-4586-479f-b01b-7adae0c869f4-kube-api-access-q75rs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:25.361407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.361246 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8svp\" (UniqueName: \"kubernetes.io/projected/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9-kube-api-access-n8svp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:25.361407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.361255 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fce9f56-4586-479f-b01b-7adae0c869f4-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:25.368842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.368816 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m"] Nov 02 14:08:25.375058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.375032 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17a7ba-6bbbddd877-5s48m"] Nov 02 14:08:25.417662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.417630 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz"] Nov 02 14:08:25.420470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.420444 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60ad62-666f48b9b7-9psbz"] Nov 02 14:08:25.431742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.431672 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fd5322-ed09-4fb3-87b1-e27d578a7af6" path="/var/lib/kubelet/pods/15fd5322-ed09-4fb3-87b1-e27d578a7af6/volumes" Nov 02 14:08:25.432230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.432209 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fce9f56-4586-479f-b01b-7adae0c869f4" path="/var/lib/kubelet/pods/4fce9f56-4586-479f-b01b-7adae0c869f4/volumes" Nov 02 14:08:25.469179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.469146 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82"] Nov 02 14:08:25.471723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.471703 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-17a7ba-58b8887899-c6c82"] Nov 02 14:08:25.541926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.541864 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:25.562151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.557391 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x"] Nov 02 14:08:25.562151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:25.557445 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60ad62-6bc99f9d58-ftg2x"] Nov 02 14:08:27.310849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:27.310808 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:27.433179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:27.433138 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6caca4-7449-4a83-b2b1-7747ff0e2fc9" path="/var/lib/kubelet/pods/cf6caca4-7449-4a83-b2b1-7747ff0e2fc9/volumes" Nov 02 14:08:27.433644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:27.433616 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa0bbfa-7b4f-46b6-8850-7e425d9d2a66" path="/var/lib/kubelet/pods/efa0bbfa-7b4f-46b6-8850-7e425d9d2a66/volumes" Nov 02 14:08:27.926002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:27.921951 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:28.110580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:28.110516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:28.692716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:28.692681 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:08:28.719545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:28.719517 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:28.914153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:28.914111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:29.200008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:29.183347 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:29.390024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:29.389781 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:29.685027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:29.683080 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:30.011673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:30.011592 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx"] Nov 02 14:08:30.012144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:30.011869 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" containerID="cri-o://a774cc9237a4843827c2e00371cad31b8457bad82ad267d19155a91495fe9387" gracePeriod=30 Nov 02 14:08:30.144001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:30.143942 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:30.227192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:30.227157 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:08:30.688976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:30.688939 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:08:30.689158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:30.689014 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:08:30.853009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:30.851854 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:31.084934 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:31.084843 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:08:31.484011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:31.476506 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:33.190584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:33.190490 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:34.862306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:34.862256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:35.063188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.060837 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:35.263014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.262489 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:35.431207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.431171 5591 generic.go:356] "Generic (PLEG): container finished" podID="7833de90-13be-4b90-a950-1d3083a72f92" containerID="a774cc9237a4843827c2e00371cad31b8457bad82ad267d19155a91495fe9387" exitCode=0 Nov 02 14:08:35.432592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.432561 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" event={"ID":"7833de90-13be-4b90-a950-1d3083a72f92","Type":"ContainerDied","Data":"a774cc9237a4843827c2e00371cad31b8457bad82ad267d19155a91495fe9387"} Nov 02 14:08:35.432760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.432621 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" event={"ID":"7833de90-13be-4b90-a950-1d3083a72f92","Type":"ContainerDied","Data":"0c9cedbf67ae9c0c5f8a22cc7ea2c1ee2d285475ec2ff2c83bd7d15389e77011"} Nov 02 14:08:35.432760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.432640 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9cedbf67ae9c0c5f8a22cc7ea2c1ee2d285475ec2ff2c83bd7d15389e77011" Nov 02 14:08:35.432760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.432654 5591 scope.go:122] "RemoveContainer" containerID="979b5222ffb680c54a7d289f7bf416038308347e89358165da85caecac355d7b" Nov 02 14:08:35.434040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.434014 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:08:35.483174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.483145 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv"] Nov 02 14:08:35.503295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503266 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr"] Nov 02 14:08:35.503466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503445 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-kube-api-access-l5dw2\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-kube-api-access-l5dw2\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.503581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503564 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-socket\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.503658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503643 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-credential-socket\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.503740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503686 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-istio-token\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-istio-token\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.503797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503747 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7833de90-13be-4b90-a950-1d3083a72f92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7833de90-13be-4b90-a950-1d3083a72f92-istiod-ca-cert\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.503797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503771 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7833de90-13be-4b90-a950-1d3083a72f92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7833de90-13be-4b90-a950-1d3083a72f92-istio-podinfo\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.503902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503822 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-data\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.503902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503861 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-certs\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.503902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503895 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-envoy\") pod \"7833de90-13be-4b90-a950-1d3083a72f92\" (UID: \"7833de90-13be-4b90-a950-1d3083a72f92\") " Nov 02 14:08:35.504074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503917 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-socket" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:35.504074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.503928 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-credential-socket" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:35.504187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.504156 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-data" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:35.504187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.504174 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-certs" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:35.504295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.504194 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7833de90-13be-4b90-a950-1d3083a72f92-istiod-ca-cert" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:35.504671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.504645 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.504671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.504672 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.504853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.504688 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7833de90-13be-4b90-a950-1d3083a72f92-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.504853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.504702 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.504853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.504716 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.506354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.506326 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-kube-api-access-l5dw2" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "kube-api-access-l5dw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:35.506547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.506521 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-istio-token" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:35.506547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.506531 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7833de90-13be-4b90-a950-1d3083a72f92-istio-podinfo" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:08:35.506674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.506569 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-envoy" pod "7833de90-13be-4b90-a950-1d3083a72f92" (UID: "7833de90-13be-4b90-a950-1d3083a72f92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:08:35.606847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.606802 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5dw2\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-kube-api-access-l5dw2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.606847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.606846 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7833de90-13be-4b90-a950-1d3083a72f92-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.607118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.606862 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7833de90-13be-4b90-a950-1d3083a72f92-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.607118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.606876 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7833de90-13be-4b90-a950-1d3083a72f92-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:35.667525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.667432 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:35.885213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:35.885179 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:36.439923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:36.439881 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx" Nov 02 14:08:36.549247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:36.549216 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx"] Nov 02 14:08:36.551843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:36.551819 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-17a7ba-openshift-default-645d755c-k25rx"] Nov 02 14:08:36.890370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:36.890310 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:37.433119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:37.433086 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7833de90-13be-4b90-a950-1d3083a72f92" path="/var/lib/kubelet/pods/7833de90-13be-4b90-a950-1d3083a72f92/volumes" Nov 02 14:08:37.824859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:37.824745 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:39.187361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:39.186126 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:39.467859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:39.467782 5591 generic.go:356] "Generic (PLEG): container finished" podID="ea174eb3-9111-4203-ac02-d7156399a40f" containerID="69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd" exitCode=0 Nov 02 14:08:39.467859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:39.467849 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" event={"ID":"ea174eb3-9111-4203-ac02-d7156399a40f","Type":"ContainerDied","Data":"69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd"} Nov 02 14:08:39.468320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:39.468301 5591 scope.go:122] "RemoveContainer" containerID="69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd" Nov 02 14:08:39.469778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:39.469744 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:39.469871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:39.469816 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:39.469871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:39.469849 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:40.476488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.476454 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" event={"ID":"ea174eb3-9111-4203-ac02-d7156399a40f","Type":"ContainerStarted","Data":"eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5"} Nov 02 14:08:40.483943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.483907 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" podUID="8749ee23-a700-4b18-b691-ecd026e34239" containerName="payload-processing" containerID="cri-o://c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57" gracePeriod=25 Nov 02 14:08:40.504262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.504224 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" podUID="809a5f84-4c45-48d1-81b1-926f8e250dba" containerName="payload-pre-processing" containerID="cri-o://42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa" gracePeriod=25 Nov 02 14:08:40.562533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.562502 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:40.815175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.815147 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:08:40.893703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.893676 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:08:40.909585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.909548 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8749ee23-a700-4b18-b691-ecd026e34239-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8749ee23-a700-4b18-b691-ecd026e34239-plugins-config-volume\") pod \"8749ee23-a700-4b18-b691-ecd026e34239\" (UID: \"8749ee23-a700-4b18-b691-ecd026e34239\") " Nov 02 14:08:40.909821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.909803 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8749ee23-a700-4b18-b691-ecd026e34239-kube-api-access-f9bbf\" (UniqueName: \"kubernetes.io/projected/8749ee23-a700-4b18-b691-ecd026e34239-kube-api-access-f9bbf\") pod \"8749ee23-a700-4b18-b691-ecd026e34239\" (UID: \"8749ee23-a700-4b18-b691-ecd026e34239\") " Nov 02 14:08:40.910043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.910015 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8749ee23-a700-4b18-b691-ecd026e34239-plugins-config-volume" pod "8749ee23-a700-4b18-b691-ecd026e34239" (UID: "8749ee23-a700-4b18-b691-ecd026e34239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:40.910874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.910844 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8749ee23-a700-4b18-b691-ecd026e34239-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:40.913137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.913107 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn"] Nov 02 14:08:40.914860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.914835 5591 cpu_manager.go:414] "containerMap: removing container" podUID="efa0bbfa-7b4f-46b6-8850-7e425d9d2a66" containerName="payload-pre-processing" Nov 02 14:08:40.914992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.914861 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa0bbfa-7b4f-46b6-8850-7e425d9d2a66" containerName="payload-pre-processing" Nov 02 14:08:40.914992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.914885 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4fce9f56-4586-479f-b01b-7adae0c869f4" containerName="payload-pre-processing" Nov 02 14:08:40.914992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.914894 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fce9f56-4586-479f-b01b-7adae0c869f4" containerName="payload-pre-processing" Nov 02 14:08:40.914992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.914937 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" Nov 02 14:08:40.914992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.914945 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915017 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8749ee23-a700-4b18-b691-ecd026e34239" containerName="payload-processing" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915026 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8749ee23-a700-4b18-b691-ecd026e34239" containerName="payload-processing" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915048 5591 cpu_manager.go:414] "containerMap: removing container" podUID="cf6caca4-7449-4a83-b2b1-7747ff0e2fc9" containerName="payload-processing" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915056 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6caca4-7449-4a83-b2b1-7747ff0e2fc9" containerName="payload-processing" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915092 5591 cpu_manager.go:414] "containerMap: removing container" podUID="15fd5322-ed09-4fb3-87b1-e27d578a7af6" containerName="maas-api" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915100 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fd5322-ed09-4fb3-87b1-e27d578a7af6" containerName="maas-api" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915131 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915139 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915160 5591 cpu_manager.go:414] "containerMap: removing container" podUID="809a5f84-4c45-48d1-81b1-926f8e250dba" containerName="payload-pre-processing" Nov 02 14:08:40.915240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915168 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809a5f84-4c45-48d1-81b1-926f8e250dba" containerName="payload-pre-processing" Nov 02 14:08:40.915691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915431 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa0bbfa-7b4f-46b6-8850-7e425d9d2a66" containerName="payload-pre-processing" Nov 02 14:08:40.915691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915447 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fce9f56-4586-479f-b01b-7adae0c869f4" containerName="payload-pre-processing" Nov 02 14:08:40.915691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915477 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" Nov 02 14:08:40.915691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915499 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8749ee23-a700-4b18-b691-ecd026e34239" containerName="payload-processing" Nov 02 14:08:40.915691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915515 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6caca4-7449-4a83-b2b1-7747ff0e2fc9" containerName="payload-processing" Nov 02 14:08:40.915691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915534 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fd5322-ed09-4fb3-87b1-e27d578a7af6" containerName="maas-api" Nov 02 14:08:40.915691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.915545 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="809a5f84-4c45-48d1-81b1-926f8e250dba" containerName="payload-pre-processing" Nov 02 14:08:40.919635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.919597 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8749ee23-a700-4b18-b691-ecd026e34239-kube-api-access-f9bbf" pod "8749ee23-a700-4b18-b691-ecd026e34239" (UID: "8749ee23-a700-4b18-b691-ecd026e34239"). InnerVolumeSpecName "kube-api-access-f9bbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:40.920389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.920368 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:40.960012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.956546 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b7d400-serving-cert\"" Nov 02 14:08:40.986032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:40.984916 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn"] Nov 02 14:08:41.011708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.011673 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809a5f84-4c45-48d1-81b1-926f8e250dba-kube-api-access-ph6xh\" (UniqueName: \"kubernetes.io/projected/809a5f84-4c45-48d1-81b1-926f8e250dba-kube-api-access-ph6xh\") pod \"809a5f84-4c45-48d1-81b1-926f8e250dba\" (UID: \"809a5f84-4c45-48d1-81b1-926f8e250dba\") " Nov 02 14:08:41.011888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.011776 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809a5f84-4c45-48d1-81b1-926f8e250dba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809a5f84-4c45-48d1-81b1-926f8e250dba-plugins-config-volume\") pod \"809a5f84-4c45-48d1-81b1-926f8e250dba\" (UID: \"809a5f84-4c45-48d1-81b1-926f8e250dba\") " Nov 02 14:08:41.012266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.012237 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chg4z\" (UniqueName: \"kubernetes.io/projected/b8a9f158-2322-41bd-a21c-dd994ad10e74-kube-api-access-chg4z\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.012381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.012303 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a9f158-2322-41bd-a21c-dd994ad10e74-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.012450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.012389 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.012450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.012434 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809a5f84-4c45-48d1-81b1-926f8e250dba-plugins-config-volume" pod "809a5f84-4c45-48d1-81b1-926f8e250dba" (UID: "809a5f84-4c45-48d1-81b1-926f8e250dba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:41.012631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.012463 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-metrics-certs\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.012631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.012545 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9bbf\" (UniqueName: \"kubernetes.io/projected/8749ee23-a700-4b18-b691-ecd026e34239-kube-api-access-f9bbf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:41.012631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.012619 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809a5f84-4c45-48d1-81b1-926f8e250dba-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:41.015049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.014460 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809a5f84-4c45-48d1-81b1-926f8e250dba-kube-api-access-ph6xh" pod "809a5f84-4c45-48d1-81b1-926f8e250dba" (UID: "809a5f84-4c45-48d1-81b1-926f8e250dba"). InnerVolumeSpecName "kube-api-access-ph6xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:41.023453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.023428 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8"] Nov 02 14:08:41.024126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.024110 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7833de90-13be-4b90-a950-1d3083a72f92" containerName="istio-proxy" Nov 02 14:08:41.027794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.027734 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.062529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.060940 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-cc8d39-openshift-default-dockercfg-sn5mg\"" Nov 02 14:08:41.069089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.069053 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8"] Nov 02 14:08:41.114646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.114606 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.114808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.114733 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-metrics-certs\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.114808 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:41.114771 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-serving-cert: secret "maas-api-e2e-praxis-a-b7d400-serving-cert" not found Nov 02 14:08:41.114903 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:41.114887 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls podName:b8a9f158-2322-41bd-a21c-dd994ad10e74 nodeName:}" failed. No retries permitted until 2026-11-02 14:08:41.614851624 +0000 UTC m=+1470.340473375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls") pod "maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" (UID: "b8a9f158-2322-41bd-a21c-dd994ad10e74") : secret "maas-api-e2e-praxis-a-b7d400-serving-cert" not found Nov 02 14:08:41.114948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.114917 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chg4z\" (UniqueName: \"kubernetes.io/projected/b8a9f158-2322-41bd-a21c-dd994ad10e74-kube-api-access-chg4z\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.115036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.115019 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a9f158-2322-41bd-a21c-dd994ad10e74-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.115169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.115153 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph6xh\" (UniqueName: \"kubernetes.io/projected/809a5f84-4c45-48d1-81b1-926f8e250dba-kube-api-access-ph6xh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:41.116202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.116175 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a9f158-2322-41bd-a21c-dd994ad10e74-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.117573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.117554 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-metrics-certs\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.156602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.156554 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:41.157035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.156969 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:08:41.157830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.157799 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chg4z\" (UniqueName: \"kubernetes.io/projected/b8a9f158-2322-41bd-a21c-dd994ad10e74-kube-api-access-chg4z\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.157968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.157891 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 14:08:41.158060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.158037 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 14:08:41.219005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.216894 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qbh\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-kube-api-access-88qbh\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.219005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.217017 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-data\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.219005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.217054 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b8850-3749-47d0-b47e-f77f649f77d3-istiod-ca-cert\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.219005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.217242 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-socket\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.219005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.217275 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-token\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.219005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.217301 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-envoy\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.219005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.217403 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-certs\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.223595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.217463 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-podinfo\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.223595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.223136 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-credential-socket\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.331555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.331459 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-podinfo\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.346862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.346827 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-credential-socket\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.347287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.347201 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qbh\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-kube-api-access-88qbh\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.347901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.347881 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-data\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.348538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.348519 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b8850-3749-47d0-b47e-f77f649f77d3-istiod-ca-cert\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.354700 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-socket\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.354744 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-token\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.354769 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-envoy\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.354885 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-certs\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.355273 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-certs\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.337807 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-podinfo\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.348304 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-data\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.347771 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-credential-socket\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.354433 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b8850-3749-47d0-b47e-f77f649f77d3-istiod-ca-cert\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.357020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.355697 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-socket\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.371003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.363968 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-envoy\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.389022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.378842 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:41.412006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.397746 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-token\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.412006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.402706 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qbh\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-kube-api-access-88qbh\") pod \"e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.486805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.486772 5591 generic.go:356] "Generic (PLEG): container finished" podID="8749ee23-a700-4b18-b691-ecd026e34239" containerID="c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57" exitCode=0 Nov 02 14:08:41.487115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.486826 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" Nov 02 14:08:41.487115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.486890 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" event={"ID":"8749ee23-a700-4b18-b691-ecd026e34239","Type":"ContainerDied","Data":"c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57"} Nov 02 14:08:41.487115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.486944 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv" event={"ID":"8749ee23-a700-4b18-b691-ecd026e34239","Type":"ContainerDied","Data":"5eb84c0effb0f34802a4bfc940cb6bbc943e1c43e19edda5ffa371db30c41614"} Nov 02 14:08:41.487115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.487004 5591 scope.go:122] "RemoveContainer" containerID="c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57" Nov 02 14:08:41.488687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.488654 5591 generic.go:356] "Generic (PLEG): container finished" podID="809a5f84-4c45-48d1-81b1-926f8e250dba" containerID="42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa" exitCode=0 Nov 02 14:08:41.488808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.488717 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" event={"ID":"809a5f84-4c45-48d1-81b1-926f8e250dba","Type":"ContainerDied","Data":"42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa"} Nov 02 14:08:41.488808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.488746 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" Nov 02 14:08:41.488808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.488768 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr" event={"ID":"809a5f84-4c45-48d1-81b1-926f8e250dba","Type":"ContainerDied","Data":"c4eeb1df5a7fe7739cdff43590e97ffeee229f41791928c6c8531c0a81e36293"} Nov 02 14:08:41.513698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.513674 5591 scope.go:122] "RemoveContainer" containerID="c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57" Nov 02 14:08:41.514062 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:41.514039 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57\": container with ID starting with c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57 not found: ID does not exist" containerID="c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57" Nov 02 14:08:41.514121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.514073 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57"} err="failed to get container status \"c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57\": rpc error: code = NotFound desc = could not find container \"c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57\": container with ID starting with c3aad3281f7bb85d69d8410a68eee6b274a36cfc0a2c434f7e049cbb0f953b57 not found: ID does not exist" Nov 02 14:08:41.514121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.514090 5591 scope.go:122] "RemoveContainer" containerID="42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa" Nov 02 14:08:41.524084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.524055 5591 scope.go:122] "RemoveContainer" containerID="42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa" Nov 02 14:08:41.524495 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:41.524458 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa\": container with ID starting with 42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa not found: ID does not exist" containerID="42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa" Nov 02 14:08:41.524656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.524633 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa"} err="failed to get container status \"42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa\": rpc error: code = NotFound desc = could not find container \"42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa\": container with ID starting with 42e10f967be942eda14d00f1ff839e06087c671c730aa1d289ddb53d393862fa not found: ID does not exist" Nov 02 14:08:41.562229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.562194 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv"] Nov 02 14:08:41.568775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.568748 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-cd99f7cdc-dznsv"] Nov 02 14:08:41.606890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.606820 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr"] Nov 02 14:08:41.609281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.609260 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-6ccdc8cc4c-zr8lr"] Nov 02 14:08:41.656329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.656257 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:41.670765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.670736 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.672945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.672920 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.887556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.887521 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:41.997340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:41.997303 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8"] Nov 02 14:08:42.015654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.014431 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:42.015654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.014522 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:42.015654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.014565 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:08:42.162375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.162159 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 14:08:42.162375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.162219 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 14:08:42.295265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.294070 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn"] Nov 02 14:08:42.498132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.498048 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" event={"ID":"b8a9f158-2322-41bd-a21c-dd994ad10e74","Type":"ContainerStarted","Data":"d77abf630d83a4c17080bbfa8678653aea9fbbe13bc43a13357e41db1e5129b0"} Nov 02 14:08:42.499465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.499439 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" event={"ID":"ce9b8850-3749-47d0-b47e-f77f649f77d3","Type":"ContainerStarted","Data":"7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a"} Nov 02 14:08:42.499604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.499487 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" event={"ID":"ce9b8850-3749-47d0-b47e-f77f649f77d3","Type":"ContainerStarted","Data":"0c4b744430cc66fda86dbf71049786a6e93ce75536a78c28b307b674836487a8"} Nov 02 14:08:42.513288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.513263 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:08:42.673580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.673228 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:42.820312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.820262 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" podStartSLOduration=2.820245901 podStartE2EDuration="2.820245901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:42.562402807 +0000 UTC m=+1471.288024589" watchObservedRunningTime="2026-11-02 14:08:42.820245901 +0000 UTC m=+1471.545867670" Nov 02 14:08:42.822070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.822044 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2"] Nov 02 14:08:42.822390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.822370 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" podUID="4493bf10-287d-48d6-8b88-9cbf447e9496" containerName="payload-processing" containerID="cri-o://134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8" gracePeriod=30 Nov 02 14:08:42.824394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.824366 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:42.868327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.868291 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2"] Nov 02 14:08:42.868582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.868548 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" podUID="b69dd95b-ae85-40f1-8b41-a5c29a2a89ac" containerName="payload-pre-processing" containerID="cri-o://53192e5c0494562ecc83dbd3537a96c04d851af2570051399365c6bac1df0f4f" gracePeriod=30 Nov 02 14:08:42.994871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:42.994812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:43.143011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.140662 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 14:08:43.143011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.140721 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 14:08:43.573145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.572942 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809a5f84-4c45-48d1-81b1-926f8e250dba" path="/var/lib/kubelet/pods/809a5f84-4c45-48d1-81b1-926f8e250dba/volumes" Nov 02 14:08:43.573921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.573645 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8749ee23-a700-4b18-b691-ecd026e34239" path="/var/lib/kubelet/pods/8749ee23-a700-4b18-b691-ecd026e34239/volumes" Nov 02 14:08:43.577853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.577829 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:08:43.577975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.577934 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" event={"ID":"b8a9f158-2322-41bd-a21c-dd994ad10e74","Type":"ContainerStarted","Data":"c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b"} Nov 02 14:08:43.578129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.578113 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:43.579728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.579705 5591 generic.go:356] "Generic (PLEG): container finished" podID="b69dd95b-ae85-40f1-8b41-a5c29a2a89ac" containerID="53192e5c0494562ecc83dbd3537a96c04d851af2570051399365c6bac1df0f4f" exitCode=0 Nov 02 14:08:43.579829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.579813 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" event={"ID":"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac","Type":"ContainerDied","Data":"53192e5c0494562ecc83dbd3537a96c04d851af2570051399365c6bac1df0f4f"} Nov 02 14:08:43.579904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.579858 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" event={"ID":"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac","Type":"ContainerDied","Data":"36c1ca517aab54bbd64367fedf9a52c56d64fa58eeb34908d4228111ca34500d"} Nov 02 14:08:43.579904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.579884 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36c1ca517aab54bbd64367fedf9a52c56d64fa58eeb34908d4228111ca34500d" Nov 02 14:08:43.581163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.581144 5591 generic.go:356] "Generic (PLEG): container finished" podID="4493bf10-287d-48d6-8b88-9cbf447e9496" containerID="134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8" exitCode=0 Nov 02 14:08:43.581236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.581174 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" event={"ID":"4493bf10-287d-48d6-8b88-9cbf447e9496","Type":"ContainerDied","Data":"134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8"} Nov 02 14:08:43.581236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.581228 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" event={"ID":"4493bf10-287d-48d6-8b88-9cbf447e9496","Type":"ContainerDied","Data":"0a28ac03609eb4f7cf5ffc3b189eaf910a1044cb2e4d180c0d3c2f63ed2b131c"} Nov 02 14:08:43.581341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.581258 5591 scope.go:122] "RemoveContainer" containerID="134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8" Nov 02 14:08:43.581341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.581184 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2" Nov 02 14:08:43.594106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.594089 5591 scope.go:122] "RemoveContainer" containerID="134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8" Nov 02 14:08:43.594344 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:08:43.594323 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8\": container with ID starting with 134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8 not found: ID does not exist" containerID="134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8" Nov 02 14:08:43.594414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.594354 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8"} err="failed to get container status \"134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8\": rpc error: code = NotFound desc = could not find container \"134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8\": container with ID starting with 134e6e9d05fa743a0990d6880565d23ea70c1c0deef6cf4376fc27bb20d0dcf8 not found: ID does not exist" Nov 02 14:08:43.595172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.595159 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:08:43.686411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.686307 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:43.728008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.706344 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:43.755024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.751330 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-plugins-config-volume\") pod \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\" (UID: \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\") " Nov 02 14:08:43.755024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.751429 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4493bf10-287d-48d6-8b88-9cbf447e9496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4493bf10-287d-48d6-8b88-9cbf447e9496-plugins-config-volume\") pod \"4493bf10-287d-48d6-8b88-9cbf447e9496\" (UID: \"4493bf10-287d-48d6-8b88-9cbf447e9496\") " Nov 02 14:08:43.755024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.751541 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4493bf10-287d-48d6-8b88-9cbf447e9496-kube-api-access-m5m7s\" (UniqueName: \"kubernetes.io/projected/4493bf10-287d-48d6-8b88-9cbf447e9496-kube-api-access-m5m7s\") pod \"4493bf10-287d-48d6-8b88-9cbf447e9496\" (UID: \"4493bf10-287d-48d6-8b88-9cbf447e9496\") " Nov 02 14:08:43.755024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.751773 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-kube-api-access-mpk8z\" (UniqueName: \"kubernetes.io/projected/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-kube-api-access-mpk8z\") pod \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\" (UID: \"b69dd95b-ae85-40f1-8b41-a5c29a2a89ac\") " Nov 02 14:08:43.755024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.753262 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-plugins-config-volume" pod "b69dd95b-ae85-40f1-8b41-a5c29a2a89ac" (UID: "b69dd95b-ae85-40f1-8b41-a5c29a2a89ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:43.755024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.753680 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4493bf10-287d-48d6-8b88-9cbf447e9496-plugins-config-volume" pod "4493bf10-287d-48d6-8b88-9cbf447e9496" (UID: "4493bf10-287d-48d6-8b88-9cbf447e9496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:08:43.770496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.770464 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4493bf10-287d-48d6-8b88-9cbf447e9496-kube-api-access-m5m7s" pod "4493bf10-287d-48d6-8b88-9cbf447e9496" (UID: "4493bf10-287d-48d6-8b88-9cbf447e9496"). InnerVolumeSpecName "kube-api-access-m5m7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:43.771326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.771299 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-kube-api-access-mpk8z" pod "b69dd95b-ae85-40f1-8b41-a5c29a2a89ac" (UID: "b69dd95b-ae85-40f1-8b41-a5c29a2a89ac"). InnerVolumeSpecName "kube-api-access-mpk8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:08:43.818945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.818899 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" podStartSLOduration=3.375193911 podStartE2EDuration="3.818885316s" totalImagesPullingTime="443.691405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:43.750468095 +0000 UTC m=+1472.476089865" watchObservedRunningTime="2026-11-02 14:08:43.818885316 +0000 UTC m=+1472.544507086" Nov 02 14:08:43.855003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.854241 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpk8z\" (UniqueName: \"kubernetes.io/projected/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-kube-api-access-mpk8z\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:43.855003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.854298 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:43.855003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.854315 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4493bf10-287d-48d6-8b88-9cbf447e9496-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:43.855003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:43.854331 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5m7s\" (UniqueName: \"kubernetes.io/projected/4493bf10-287d-48d6-8b88-9cbf447e9496-kube-api-access-m5m7s\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:08:44.015425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.015389 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2"] Nov 02 14:08:44.018851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.018824 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7489f-cfb9f8456-rgsz2"] Nov 02 14:08:44.151019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.137928 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 14:08:44.151019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.138008 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 14:08:44.588485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.588453 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2" Nov 02 14:08:44.588901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.588611 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:44.589694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.589668 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:08:44.657650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.657602 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2"] Nov 02 14:08:44.660854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:44.660818 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-cb546f4c-8jkm2"] Nov 02 14:08:45.152501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:45.152459 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 14:08:45.152788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:45.152760 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 14:08:45.449594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:45.449510 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4493bf10-287d-48d6-8b88-9cbf447e9496" path="/var/lib/kubelet/pods/4493bf10-287d-48d6-8b88-9cbf447e9496/volumes" Nov 02 14:08:45.450534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:45.450504 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69dd95b-ae85-40f1-8b41-a5c29a2a89ac" path="/var/lib/kubelet/pods/b69dd95b-ae85-40f1-8b41-a5c29a2a89ac/volumes" Nov 02 14:08:45.504728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:45.493254 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:45.708637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:45.708552 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:46.139396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:46.139367 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:08:46.140372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:46.140350 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:08:46.262484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:46.262423 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:46.479219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:46.479126 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:47.247170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.247133 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:47.313852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.313819 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j"] Nov 02 14:08:47.314464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.314449 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b69dd95b-ae85-40f1-8b41-a5c29a2a89ac" containerName="payload-pre-processing" Nov 02 14:08:47.314515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.314466 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69dd95b-ae85-40f1-8b41-a5c29a2a89ac" containerName="payload-pre-processing" Nov 02 14:08:47.314515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.314497 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4493bf10-287d-48d6-8b88-9cbf447e9496" containerName="payload-processing" Nov 02 14:08:47.314515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.314502 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4493bf10-287d-48d6-8b88-9cbf447e9496" containerName="payload-processing" Nov 02 14:08:47.314604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.314599 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69dd95b-ae85-40f1-8b41-a5c29a2a89ac" containerName="payload-pre-processing" Nov 02 14:08:47.314634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.314610 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4493bf10-287d-48d6-8b88-9cbf447e9496" containerName="payload-processing" Nov 02 14:08:47.319303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.319277 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:47.349091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.349062 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-cc8d39\"" Nov 02 14:08:47.353686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.353661 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j"] Nov 02 14:08:47.415042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.415009 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2"] Nov 02 14:08:47.419123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.419091 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwpxm\" (UniqueName: \"kubernetes.io/projected/f3da4df8-c04e-42aa-bb51-df99e073b242-kube-api-access-rwpxm\") pod \"payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j\" (UID: \"f3da4df8-c04e-42aa-bb51-df99e073b242\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:47.419280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.419176 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3da4df8-c04e-42aa-bb51-df99e073b242-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j\" (UID: \"f3da4df8-c04e-42aa-bb51-df99e073b242\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:47.419280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.419214 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:47.442377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.442142 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-cc8d39-dockercfg-rrz65\"" Nov 02 14:08:47.445668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.445639 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2"] Nov 02 14:08:47.458593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.458549 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:47.524220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.522467 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwpxm\" (UniqueName: \"kubernetes.io/projected/f3da4df8-c04e-42aa-bb51-df99e073b242-kube-api-access-rwpxm\") pod \"payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j\" (UID: \"f3da4df8-c04e-42aa-bb51-df99e073b242\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:47.524220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.522543 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3da4df8-c04e-42aa-bb51-df99e073b242-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j\" (UID: \"f3da4df8-c04e-42aa-bb51-df99e073b242\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:47.524220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.523586 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3da4df8-c04e-42aa-bb51-df99e073b242-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j\" (UID: \"f3da4df8-c04e-42aa-bb51-df99e073b242\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:47.605915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.605848 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwpxm\" (UniqueName: \"kubernetes.io/projected/f3da4df8-c04e-42aa-bb51-df99e073b242-kube-api-access-rwpxm\") pod \"payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j\" (UID: \"f3da4df8-c04e-42aa-bb51-df99e073b242\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:47.650662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.648843 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:47.650662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.650325 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:47.688648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.682416 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d30ac92-1b49-41e0-90fa-fe108d5679b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2\" (UID: \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:47.688648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.682718 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlpv\" (UniqueName: \"kubernetes.io/projected/6d30ac92-1b49-41e0-90fa-fe108d5679b8-kube-api-access-rmlpv\") pod \"payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2\" (UID: \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:47.784442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.784110 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d30ac92-1b49-41e0-90fa-fe108d5679b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2\" (UID: \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:47.784442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.784269 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlpv\" (UniqueName: \"kubernetes.io/projected/6d30ac92-1b49-41e0-90fa-fe108d5679b8-kube-api-access-rmlpv\") pod \"payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2\" (UID: \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:47.785617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.785359 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d30ac92-1b49-41e0-90fa-fe108d5679b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2\" (UID: \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:47.811458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:47.811425 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlpv\" (UniqueName: \"kubernetes.io/projected/6d30ac92-1b49-41e0-90fa-fe108d5679b8-kube-api-access-rmlpv\") pod \"payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2\" (UID: \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:48.036701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.036674 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j"] Nov 02 14:08:48.037968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.037935 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:48.038254 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:08:48.038229 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3da4df8_c04e_42aa_bb51_df99e073b242.slice/crio-f67a9be0ef68d11a1b9239c7788bf8de279e2d8107911c0b7aa83b979d7b86da WatchSource:0}: Error finding container f67a9be0ef68d11a1b9239c7788bf8de279e2d8107911c0b7aa83b979d7b86da: Status 404 returned error can't find the container with id f67a9be0ef68d11a1b9239c7788bf8de279e2d8107911c0b7aa83b979d7b86da Nov 02 14:08:48.345084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.345057 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2"] Nov 02 14:08:48.345712 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:08:48.345669 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d30ac92_1b49_41e0_90fa_fe108d5679b8.slice/crio-694b948aa68e0de0cf67c38050142e5884974ed3d73cdbcbf272faf7446f5830 WatchSource:0}: Error finding container 694b948aa68e0de0cf67c38050142e5884974ed3d73cdbcbf272faf7446f5830: Status 404 returned error can't find the container with id 694b948aa68e0de0cf67c38050142e5884974ed3d73cdbcbf272faf7446f5830 Nov 02 14:08:48.742286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.742251 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" event={"ID":"6d30ac92-1b49-41e0-90fa-fe108d5679b8","Type":"ContainerStarted","Data":"95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440"} Nov 02 14:08:48.742472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.742307 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" event={"ID":"6d30ac92-1b49-41e0-90fa-fe108d5679b8","Type":"ContainerStarted","Data":"694b948aa68e0de0cf67c38050142e5884974ed3d73cdbcbf272faf7446f5830"} Nov 02 14:08:48.742472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.742328 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:48.743842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.743820 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" event={"ID":"f3da4df8-c04e-42aa-bb51-df99e073b242","Type":"ContainerStarted","Data":"2e22581b56688628061f40bf7b864c21975c700763186386654515b2819cdd2c"} Nov 02 14:08:48.743969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.743860 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" event={"ID":"f3da4df8-c04e-42aa-bb51-df99e073b242","Type":"ContainerStarted","Data":"f67a9be0ef68d11a1b9239c7788bf8de279e2d8107911c0b7aa83b979d7b86da"} Nov 02 14:08:48.743969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.743944 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:08:48.796429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.796380 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" podStartSLOduration=1.796367263 podStartE2EDuration="1.796367263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:48.794555165 +0000 UTC m=+1477.520176934" watchObservedRunningTime="2026-11-02 14:08:48.796367263 +0000 UTC m=+1477.521989032" Nov 02 14:08:48.840566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:48.840511 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" podStartSLOduration=1.840497555 podStartE2EDuration="1.840497555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:48.837799116 +0000 UTC m=+1477.563420886" watchObservedRunningTime="2026-11-02 14:08:48.840497555 +0000 UTC m=+1477.566119385" Nov 02 14:08:49.534011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:49.529025 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:49.594797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:49.594758 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:08:50.686784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:50.686748 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:50.937103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:50.921851 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:51.032011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.031952 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm"] Nov 02 14:08:51.035723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.035707 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:51.067660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.067623 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-54f2fe\"" Nov 02 14:08:51.067954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.067934 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-54f2fe-dockercfg-fdxg9\"" Nov 02 14:08:51.070974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.070953 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm"] Nov 02 14:08:51.182008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.175111 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9417156-daa1-4147-86df-f8fbadca2ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm\" (UID: \"a9417156-daa1-4147-86df-f8fbadca2ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:51.182008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.175248 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpkf\" (UniqueName: \"kubernetes.io/projected/a9417156-daa1-4147-86df-f8fbadca2ac9-kube-api-access-pbpkf\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm\" (UID: \"a9417156-daa1-4147-86df-f8fbadca2ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:51.235056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.234018 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b"] Nov 02 14:08:51.257007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.253522 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:51.272004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.268023 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b"] Nov 02 14:08:51.320302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.319033 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpkf\" (UniqueName: \"kubernetes.io/projected/a9417156-daa1-4147-86df-f8fbadca2ac9-kube-api-access-pbpkf\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm\" (UID: \"a9417156-daa1-4147-86df-f8fbadca2ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:51.320302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.319312 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9417156-daa1-4147-86df-f8fbadca2ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm\" (UID: \"a9417156-daa1-4147-86df-f8fbadca2ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:51.320573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.320324 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9417156-daa1-4147-86df-f8fbadca2ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm\" (UID: \"a9417156-daa1-4147-86df-f8fbadca2ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:51.355303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.355266 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpkf\" (UniqueName: \"kubernetes.io/projected/a9417156-daa1-4147-86df-f8fbadca2ac9-kube-api-access-pbpkf\") pod \"payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm\" (UID: \"a9417156-daa1-4147-86df-f8fbadca2ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:51.420949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.420914 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122ac438-de00-431f-829b-a2d925799aa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b\" (UID: \"122ac438-de00-431f-829b-a2d925799aa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:51.421166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.421033 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8rm\" (UniqueName: \"kubernetes.io/projected/122ac438-de00-431f-829b-a2d925799aa2-kube-api-access-kt8rm\") pod \"payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b\" (UID: \"122ac438-de00-431f-829b-a2d925799aa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:51.522028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.521905 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122ac438-de00-431f-829b-a2d925799aa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b\" (UID: \"122ac438-de00-431f-829b-a2d925799aa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:51.522224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.522033 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8rm\" (UniqueName: \"kubernetes.io/projected/122ac438-de00-431f-829b-a2d925799aa2-kube-api-access-kt8rm\") pod \"payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b\" (UID: \"122ac438-de00-431f-829b-a2d925799aa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:51.522853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.522826 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122ac438-de00-431f-829b-a2d925799aa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b\" (UID: \"122ac438-de00-431f-829b-a2d925799aa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:51.548525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.548491 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8rm\" (UniqueName: \"kubernetes.io/projected/122ac438-de00-431f-829b-a2d925799aa2-kube-api-access-kt8rm\") pod \"payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b\" (UID: \"122ac438-de00-431f-829b-a2d925799aa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:51.600926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.600897 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:51.650713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.650676 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:51.902758 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.902726 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:51.914532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.914508 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b"] Nov 02 14:08:51.915399 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:08:51.915368 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122ac438_de00_431f_829b_a2d925799aa2.slice/crio-7124f377348e5cc6b839aae8191aa0207c28f8dfc62befacb04c0df6fed1aa3d WatchSource:0}: Error finding container 7124f377348e5cc6b839aae8191aa0207c28f8dfc62befacb04c0df6fed1aa3d: Status 404 returned error can't find the container with id 7124f377348e5cc6b839aae8191aa0207c28f8dfc62befacb04c0df6fed1aa3d Nov 02 14:08:51.967146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:51.967118 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm"] Nov 02 14:08:51.970111 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:08:51.970085 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9417156_daa1_4147_86df_f8fbadca2ac9.slice/crio-04828ad8426886d3b6808625049061cebedef79304a03d50e177ff9c2262130c WatchSource:0}: Error finding container 04828ad8426886d3b6808625049061cebedef79304a03d50e177ff9c2262130c: Status 404 returned error can't find the container with id 04828ad8426886d3b6808625049061cebedef79304a03d50e177ff9c2262130c Nov 02 14:08:52.869957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:52.869914 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" event={"ID":"122ac438-de00-431f-829b-a2d925799aa2","Type":"ContainerStarted","Data":"d60a6abf62f15a21b083d5108b4aee7410560aaa0be1e922ff0e8317fc269f5d"} Nov 02 14:08:52.870207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:52.870043 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" event={"ID":"122ac438-de00-431f-829b-a2d925799aa2","Type":"ContainerStarted","Data":"7124f377348e5cc6b839aae8191aa0207c28f8dfc62befacb04c0df6fed1aa3d"} Nov 02 14:08:52.870207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:52.870079 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:08:52.872677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:52.872648 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" event={"ID":"a9417156-daa1-4147-86df-f8fbadca2ac9","Type":"ContainerStarted","Data":"3b59a0ae99f3dafe76fd77c45a0415b632ee5d67ace036ab864a6aa9d5a4a7c3"} Nov 02 14:08:52.872796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:52.872695 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" event={"ID":"a9417156-daa1-4147-86df-f8fbadca2ac9","Type":"ContainerStarted","Data":"04828ad8426886d3b6808625049061cebedef79304a03d50e177ff9c2262130c"} Nov 02 14:08:52.872796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:52.872722 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:08:52.957064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:52.956952 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" podStartSLOduration=1.956937395 podStartE2EDuration="1.956937395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:52.913588016 +0000 UTC m=+1481.639209786" watchObservedRunningTime="2026-11-02 14:08:52.956937395 +0000 UTC m=+1481.682559164" Nov 02 14:08:52.959967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:52.959922 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" podStartSLOduration=2.9599066 podStartE2EDuration="2.9599066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:52.956487171 +0000 UTC m=+1481.682108942" watchObservedRunningTime="2026-11-02 14:08:52.9599066 +0000 UTC m=+1481.685528371" Nov 02 14:08:53.754007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:53.746874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:54.541771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:54.537969 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:54.701418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:54.701387 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:55.301902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:55.301862 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:55.624063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:55.621823 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:55.865577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:55.865535 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:57.865941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:57.865835 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:58.788480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:58.788438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:59.198037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:59.197935 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:59.386045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:59.384889 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:08:59.765818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:59.765786 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:08:59.766039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:08:59.765877 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:09:00.163168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:00.163125 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:09:00.163343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:00.163208 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:09:00.452348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:00.452222 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:01.076075 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:09:01.076036 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:09:01.433457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:01.433422 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht"] Nov 02 14:09:01.574974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:01.574937 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:01.735871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:01.735786 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:02.317005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:02.316749 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:02.997052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:02.997012 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:03.003462 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.003436 5591 generic.go:356] "Generic (PLEG): container finished" podID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerID="362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199" exitCode=0 Nov 02 14:09:03.003616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.003506 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" event={"ID":"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc","Type":"ContainerDied","Data":"362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199"} Nov 02 14:09:03.003994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.003967 5591 scope.go:122] "RemoveContainer" containerID="362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199" Nov 02 14:09:03.005514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.005487 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:03.005571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.005556 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:03.005611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.005589 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:03.202549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.202516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:03.885167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.885130 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:09:03.885356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:03.885201 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:09:04.015056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:04.015015 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" event={"ID":"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc","Type":"ContainerStarted","Data":"a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738"} Nov 02 14:09:04.419643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:04.419587 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:04.792219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:04.792182 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:09:04.792423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:04.792401 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 14:09:04.792469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:04.792448 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 14:09:05.792199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:05.792152 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 14:09:05.792695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:05.792229 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 14:09:05.965246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:05.961822 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:06.197278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.197231 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:06.375217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.375172 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:06.456080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.452640 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" podUID="6ab5b908-0b0c-4f49-83c4-2a8204293fe1" containerName="payload-pre-processing" containerID="cri-o://45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9" gracePeriod=25 Nov 02 14:09:06.792411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.792381 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 14:09:06.792808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.792443 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 14:09:06.913763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.913735 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:09:06.970884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.970852 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-plugins-config-volume\") pod \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\" (UID: \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\") " Nov 02 14:09:06.971249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.971225 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-kube-api-access-688md\" (UniqueName: \"kubernetes.io/projected/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-kube-api-access-688md\") pod \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\" (UID: \"6ab5b908-0b0c-4f49-83c4-2a8204293fe1\") " Nov 02 14:09:06.971347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.971297 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-plugins-config-volume" pod "6ab5b908-0b0c-4f49-83c4-2a8204293fe1" (UID: "6ab5b908-0b0c-4f49-83c4-2a8204293fe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:09:06.972138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.972112 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:09:06.973282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:06.973259 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-kube-api-access-688md" pod "6ab5b908-0b0c-4f49-83c4-2a8204293fe1" (UID: "6ab5b908-0b0c-4f49-83c4-2a8204293fe1"). InnerVolumeSpecName "kube-api-access-688md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:09:07.065404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.065312 5591 generic.go:356] "Generic (PLEG): container finished" podID="6ab5b908-0b0c-4f49-83c4-2a8204293fe1" containerID="45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9" exitCode=0 Nov 02 14:09:07.065851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.065824 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" event={"ID":"6ab5b908-0b0c-4f49-83c4-2a8204293fe1","Type":"ContainerDied","Data":"45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9"} Nov 02 14:09:07.066044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.066024 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" event={"ID":"6ab5b908-0b0c-4f49-83c4-2a8204293fe1","Type":"ContainerDied","Data":"123a067b249f04d71fde67facb68c2b684f539db171c490d4e2bbbb1069ccc59"} Nov 02 14:09:07.066172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.066159 5591 scope.go:122] "RemoveContainer" containerID="45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9" Nov 02 14:09:07.070091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.070049 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht" Nov 02 14:09:07.086229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.081219 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-688md\" (UniqueName: \"kubernetes.io/projected/6ab5b908-0b0c-4f49-83c4-2a8204293fe1-kube-api-access-688md\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:09:07.157094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.153611 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:07.160578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.160556 5591 scope.go:122] "RemoveContainer" containerID="45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9" Nov 02 14:09:07.160912 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:09:07.160892 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9\": container with ID starting with 45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9 not found: ID does not exist" containerID="45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9" Nov 02 14:09:07.160963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.160926 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9"} err="failed to get container status \"45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9\": rpc error: code = NotFound desc = could not find container \"45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9\": container with ID starting with 45b4fb3ca6d530ea1285aa15c5ee95718e7cbe0c25b488882152a3e76bdba1b9 not found: ID does not exist" Nov 02 14:09:07.197001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.196961 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht"] Nov 02 14:09:07.202804 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.202777 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5bc4bc46cf-n4tht"] Nov 02 14:09:07.490012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.489655 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab5b908-0b0c-4f49-83c4-2a8204293fe1" path="/var/lib/kubelet/pods/6ab5b908-0b0c-4f49-83c4-2a8204293fe1/volumes" Nov 02 14:09:07.693343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.690336 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf"] Nov 02 14:09:07.693343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.691313 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6ab5b908-0b0c-4f49-83c4-2a8204293fe1" containerName="payload-pre-processing" Nov 02 14:09:07.693343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.691337 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab5b908-0b0c-4f49-83c4-2a8204293fe1" containerName="payload-pre-processing" Nov 02 14:09:07.693343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.691536 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab5b908-0b0c-4f49-83c4-2a8204293fe1" containerName="payload-pre-processing" Nov 02 14:09:07.709039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.704302 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf"] Nov 02 14:09:07.709039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.704426 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.723011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.722807 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c1d69c-openshift-default-dockercfg-zdwwp\"" Nov 02 14:09:07.783805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.783724 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:07.791775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.791748 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 14:09:07.791919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.791823 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 14:09:07.809115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809088 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-token\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.809556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809120 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5kfd\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-kube-api-access-q5kfd\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.809556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809177 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-podinfo\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.809556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809234 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-credential-socket\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.809556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809301 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-certs\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.809556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809328 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-socket\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.809556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809345 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-data\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.809556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809374 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-envoy\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.809556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.809445 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82deb4cb-c2b7-4454-82d4-51a9f3959707-istiod-ca-cert\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.926934 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-token\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.927017 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5kfd\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-kube-api-access-q5kfd\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.927103 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-podinfo\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.927188 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-credential-socket\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.927302 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-certs\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.927348 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-socket\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.927375 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-data\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.927421 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-envoy\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.927547 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82deb4cb-c2b7-4454-82d4-51a9f3959707-istiod-ca-cert\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.928425 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82deb4cb-c2b7-4454-82d4-51a9f3959707-istiod-ca-cert\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.929649 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-socket\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.929936 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-credential-socket\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.930186 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-certs\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.930419 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-data\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.940852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.940326 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-envoy\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.949379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.949313 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-podinfo\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.997369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.996607 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-token\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:07.997369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:07.997303 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5kfd\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-kube-api-access-q5kfd\") pod \"e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:08.035068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.031293 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:08.365067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.363150 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:08.567063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.564456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:08.582897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.582866 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf"] Nov 02 14:09:08.593956 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:09:08.593907 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82deb4cb_c2b7_4454_82d4_51a9f3959707.slice/crio-3a38ec0f2520ee6fcf031fdf2813958ca67ed01579423df8769446ce2e1da006 WatchSource:0}: Error finding container 3a38ec0f2520ee6fcf031fdf2813958ca67ed01579423df8769446ce2e1da006: Status 404 returned error can't find the container with id 3a38ec0f2520ee6fcf031fdf2813958ca67ed01579423df8769446ce2e1da006 Nov 02 14:09:08.603266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.599831 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:08.603266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.600145 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:08.603266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.600476 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:08.791663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.791614 5591 patch_prober.go:29] interesting pod/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 14:09:08.791866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:08.791678 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 14:09:09.189356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:09.189311 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" event={"ID":"82deb4cb-c2b7-4454-82d4-51a9f3959707","Type":"ContainerStarted","Data":"6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd"} Nov 02 14:09:09.189763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:09.189421 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" event={"ID":"82deb4cb-c2b7-4454-82d4-51a9f3959707","Type":"ContainerStarted","Data":"3a38ec0f2520ee6fcf031fdf2813958ca67ed01579423df8769446ce2e1da006"} Nov 02 14:09:09.296050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:09.295957 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" podStartSLOduration=2.295940974 podStartE2EDuration="2.295940974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:09.28857441 +0000 UTC m=+1498.014196179" watchObservedRunningTime="2026-11-02 14:09:09.295940974 +0000 UTC m=+1498.021562744" Nov 02 14:09:09.796273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:09.796235 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:09:09.796552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:09.796531 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:09:09.797180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:09.797158 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:09:09.998884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:09.998855 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:10.032282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:10.032249 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:10.038087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:10.038057 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:10.171469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:10.171432 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:10.206262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:10.206232 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:10.213330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:10.213296 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:09:11.754289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:11.754251 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:11.797238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:11.797196 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:09:11.831664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:11.831639 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:09:12.017276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:12.017184 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:12.256999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:12.256953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:12.837665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:12.837628 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:13.457001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:13.456960 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:13.870274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:13.870182 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:14.656653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:14.656602 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:16.308007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:16.301174 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:16.663532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:16.663477 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:09:18.311014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:18.310962 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:19.537202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:19.537163 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:19.711615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:19.711576 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:19.890548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:19.888233 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj"] Nov 02 14:09:19.894691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:19.894666 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:20.099038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.098782 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6tn\" (UniqueName: \"kubernetes.io/projected/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-kube-api-access-lq6tn\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj\" (UID: \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:20.099038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.098967 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj\" (UID: \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:20.099795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.099755 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-13367a\"" Nov 02 14:09:20.100155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.100136 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-13367a-dockercfg-gcbf6\"" Nov 02 14:09:20.125809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.125770 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj"] Nov 02 14:09:20.200677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.200567 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6tn\" (UniqueName: \"kubernetes.io/projected/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-kube-api-access-lq6tn\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj\" (UID: \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:20.200870 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.200740 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj\" (UID: \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:20.201787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.201756 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj\" (UID: \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:20.225148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.225055 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6tn\" (UniqueName: \"kubernetes.io/projected/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-kube-api-access-lq6tn\") pod \"payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj\" (UID: \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:20.439654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.439623 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:20.789967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.789932 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj"] Nov 02 14:09:20.790446 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:09:20.790330 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bfd415_5fa9_4cf8_a26f_35a1262f7b1b.slice/crio-50d3acbdbaff3abe1c82d2e1368eca11f465245da503ae4074f02d8f8422154d WatchSource:0}: Error finding container 50d3acbdbaff3abe1c82d2e1368eca11f465245da503ae4074f02d8f8422154d: Status 404 returned error can't find the container with id 50d3acbdbaff3abe1c82d2e1368eca11f465245da503ae4074f02d8f8422154d Nov 02 14:09:20.790867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:20.790781 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:21.284720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:21.284685 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" event={"ID":"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b","Type":"ContainerStarted","Data":"6ed8b4f08ff732a8f6c45abf3d8b2c9f27744b962ddc72441232ddcd7fb42be4"} Nov 02 14:09:21.285037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:21.284749 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" event={"ID":"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b","Type":"ContainerStarted","Data":"50d3acbdbaff3abe1c82d2e1368eca11f465245da503ae4074f02d8f8422154d"} Nov 02 14:09:21.285037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:21.284893 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:21.327853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:21.327802 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" podStartSLOduration=2.327789326 podStartE2EDuration="2.327789326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:21.326300135 +0000 UTC m=+1510.051921905" watchObservedRunningTime="2026-11-02 14:09:21.327789326 +0000 UTC m=+1510.053411162" Nov 02 14:09:22.649845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:22.649813 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:22.873009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:22.865721 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:24.512212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:24.512168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:24.675246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:24.675210 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:24.864262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:24.864181 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:26.458018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:26.456628 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:28.066787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:28.066736 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:29.782580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:29.782518 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:31.077933 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:09:31.077860 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:09:31.388112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:31.388071 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:32.292483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:32.292454 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:09:33.285021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:33.284963 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:35.055006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:35.054832 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:35.232033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:35.231043 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:36.472835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:36.472764 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:36.876120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:36.876011 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:37.676013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:37.675968 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:38.199006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:38.195966 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:38.493860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:38.487032 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:38.735419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:38.735376 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:38.785940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:38.785842 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:09:39.149078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.149039 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2"] Nov 02 14:09:39.154458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.154432 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.179714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.179675 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2"] Nov 02 14:09:39.187680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.187650 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-de493d-openshift-default-dockercfg-xm7cc\"" Nov 02 14:09:39.255639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.255591 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-socket\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.255830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.255701 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-istio-token\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.255830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.255798 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-credential-socket\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.255917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.255839 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92939b14-a553-47be-9bd8-a5de527a88ca-istiod-ca-cert\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.255917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.255858 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-certs\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.256019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.255951 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92939b14-a553-47be-9bd8-a5de527a88ca-istio-podinfo\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.256070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.256052 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-data\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.256111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.256094 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-envoy\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.256249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.256234 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86cqz\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-kube-api-access-86cqz\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.277958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.277923 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.361878 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-socket\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.361956 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-istio-token\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.362029 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-credential-socket\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.362054 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92939b14-a553-47be-9bd8-a5de527a88ca-istiod-ca-cert\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.362079 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-certs\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.362160 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92939b14-a553-47be-9bd8-a5de527a88ca-istio-podinfo\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.362213 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-data\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.362248 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-envoy\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.362391 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86cqz\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-kube-api-access-86cqz\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.363197 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-socket\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.363714 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-credential-socket\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.394008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.364352 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92939b14-a553-47be-9bd8-a5de527a88ca-istiod-ca-cert\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.395034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.394972 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-certs\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.397110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.396752 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-data\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.407843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.407766 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92939b14-a553-47be-9bd8-a5de527a88ca-istio-podinfo\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.433029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.426487 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-istio-token\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.442160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.442124 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86cqz\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-kube-api-access-86cqz\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.505016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.497594 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:39.535117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.535048 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-envoy\") pod \"e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:39.680305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.680174 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:39.770583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:39.770548 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:40.221248 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:09:40.221219 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92939b14_a553_47be_9bd8_a5de527a88ca.slice/crio-8f09a71993f240eff3a2be759f2caaa69bde0a1d99ecab30b99c1f7ebc99e990 WatchSource:0}: Error finding container 8f09a71993f240eff3a2be759f2caaa69bde0a1d99ecab30b99c1f7ebc99e990: Status 404 returned error can't find the container with id 8f09a71993f240eff3a2be759f2caaa69bde0a1d99ecab30b99c1f7ebc99e990 Nov 02 14:09:40.223243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:40.223217 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2"] Nov 02 14:09:40.223951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:40.223923 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:40.224045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:40.224002 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:40.224045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:40.224039 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:40.443188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:40.443152 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" event={"ID":"92939b14-a553-47be-9bd8-a5de527a88ca","Type":"ContainerStarted","Data":"030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e"} Nov 02 14:09:40.443364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:40.443233 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" event={"ID":"92939b14-a553-47be-9bd8-a5de527a88ca","Type":"ContainerStarted","Data":"8f09a71993f240eff3a2be759f2caaa69bde0a1d99ecab30b99c1f7ebc99e990"} Nov 02 14:09:40.490962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:40.490855 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" podStartSLOduration=1.490839769 podStartE2EDuration="1.490839769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:40.48976046 +0000 UTC m=+1529.215382230" watchObservedRunningTime="2026-11-02 14:09:40.490839769 +0000 UTC m=+1529.216461539" Nov 02 14:09:40.772032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:40.771918 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:41.451445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.451413 5591 generic.go:356] "Generic (PLEG): container finished" podID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerID="7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a" exitCode=0 Nov 02 14:09:41.451664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.451502 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" event={"ID":"ce9b8850-3749-47d0-b47e-f77f649f77d3","Type":"ContainerDied","Data":"7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a"} Nov 02 14:09:41.452066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.452044 5591 scope.go:122] "RemoveContainer" containerID="7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a" Nov 02 14:09:41.453528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.453502 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:41.453595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.453569 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:41.453633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.453599 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:09:41.640776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.640743 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:41.790217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.790189 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:41.856912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:41.856879 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:42.465313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:42.465275 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" event={"ID":"ce9b8850-3749-47d0-b47e-f77f649f77d3","Type":"ContainerStarted","Data":"3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5"} Nov 02 14:09:42.465680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:42.465661 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:42.466723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:42.466701 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:09:42.661103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:42.661067 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:09:42.662102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:42.661663 5591 patch_prober.go:29] interesting pod/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:09:42.662102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:42.661718 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:09:43.383935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.383899 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:43.594217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.594185 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:43.656790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.656705 5591 patch_prober.go:29] interesting pod/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:09:43.656943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.656789 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:09:43.727306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.727270 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m"] Nov 02 14:09:43.756381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.756338 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:43.760179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.760152 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m"] Nov 02 14:09:43.786073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.786038 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-370905\"" Nov 02 14:09:43.847008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.807424 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-370905-dockercfg-kdv7r\"" Nov 02 14:09:43.847008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.820613 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvtx\" (UniqueName: \"kubernetes.io/projected/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-kube-api-access-dvvtx\") pod \"payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m\" (UID: \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:43.847008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.820730 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m\" (UID: \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:43.876006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.855516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:43.957070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.953427 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvtx\" (UniqueName: \"kubernetes.io/projected/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-kube-api-access-dvvtx\") pod \"payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m\" (UID: \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:43.957070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.953567 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m\" (UID: \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:43.957070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:43.954778 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m\" (UID: \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:44.105051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:44.095515 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:44.115023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:44.113571 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvtx\" (UniqueName: \"kubernetes.io/projected/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-kube-api-access-dvvtx\") pod \"payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m\" (UID: \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:44.387964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:44.387933 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:44.589606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:44.589577 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:09:44.656875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:44.656781 5591 patch_prober.go:29] interesting pod/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:09:44.656875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:44.656851 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:09:44.722169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:44.722143 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m"] Nov 02 14:09:44.723174 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:09:44.723143 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f327b74_2f0c_4d9c_9ac6_c5eaf6cbe0f6.slice/crio-af0f12da96855cca63b02c0f9cf4e64ac8be1070f2b19424c84ff199857bc004 WatchSource:0}: Error finding container af0f12da96855cca63b02c0f9cf4e64ac8be1070f2b19424c84ff199857bc004: Status 404 returned error can't find the container with id af0f12da96855cca63b02c0f9cf4e64ac8be1070f2b19424c84ff199857bc004 Nov 02 14:09:45.386495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.386443 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:45.489046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.489007 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" event={"ID":"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6","Type":"ContainerStarted","Data":"17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7"} Nov 02 14:09:45.489455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.489073 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:45.489455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.489097 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" event={"ID":"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6","Type":"ContainerStarted","Data":"af0f12da96855cca63b02c0f9cf4e64ac8be1070f2b19424c84ff199857bc004"} Nov 02 14:09:45.511683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.511648 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs"] Nov 02 14:09:45.515853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.515828 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:45.563725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.563691 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-de493d-dockercfg-nkn7w\"" Nov 02 14:09:45.565732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.565680 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" podStartSLOduration=2.565662171 podStartE2EDuration="2.565662171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:45.530839668 +0000 UTC m=+1534.256461438" watchObservedRunningTime="2026-11-02 14:09:45.565662171 +0000 UTC m=+1534.291283942" Nov 02 14:09:45.587828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.587796 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs"] Nov 02 14:09:45.608914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.608882 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-de493d\"" Nov 02 14:09:45.610380 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.610345 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:45.618303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.618275 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2k8j\" (UniqueName: \"kubernetes.io/projected/752361dc-c22c-4225-9f6e-17f1034004a2-kube-api-access-j2k8j\") pod \"payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs\" (UID: \"752361dc-c22c-4225-9f6e-17f1034004a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:45.618418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.618355 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752361dc-c22c-4225-9f6e-17f1034004a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs\" (UID: \"752361dc-c22c-4225-9f6e-17f1034004a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:45.662088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.656774 5591 patch_prober.go:29] interesting pod/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:09:45.662088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.656856 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:09:45.672063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.672026 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d"] Nov 02 14:09:45.690628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.690590 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:45.725059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.721465 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d"] Nov 02 14:09:45.780555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.747693 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584b5b2-da4b-4a4e-be79-bdf70584d59f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d\" (UID: \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:45.780998 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.780953 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2k8j\" (UniqueName: \"kubernetes.io/projected/752361dc-c22c-4225-9f6e-17f1034004a2-kube-api-access-j2k8j\") pod \"payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs\" (UID: \"752361dc-c22c-4225-9f6e-17f1034004a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:45.781180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.781157 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gxz7\" (UniqueName: \"kubernetes.io/projected/f584b5b2-da4b-4a4e-be79-bdf70584d59f-kube-api-access-9gxz7\") pod \"payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d\" (UID: \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:45.781373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.781356 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752361dc-c22c-4225-9f6e-17f1034004a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs\" (UID: \"752361dc-c22c-4225-9f6e-17f1034004a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:45.782438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.782413 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752361dc-c22c-4225-9f6e-17f1034004a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs\" (UID: \"752361dc-c22c-4225-9f6e-17f1034004a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:45.913458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.888847 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584b5b2-da4b-4a4e-be79-bdf70584d59f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d\" (UID: \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:45.913458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.889041 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gxz7\" (UniqueName: \"kubernetes.io/projected/f584b5b2-da4b-4a4e-be79-bdf70584d59f-kube-api-access-9gxz7\") pod \"payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d\" (UID: \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:45.913458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.890341 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584b5b2-da4b-4a4e-be79-bdf70584d59f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d\" (UID: \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:45.946007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.942209 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2k8j\" (UniqueName: \"kubernetes.io/projected/752361dc-c22c-4225-9f6e-17f1034004a2-kube-api-access-j2k8j\") pod \"payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs\" (UID: \"752361dc-c22c-4225-9f6e-17f1034004a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:45.994017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:45.979580 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gxz7\" (UniqueName: \"kubernetes.io/projected/f584b5b2-da4b-4a4e-be79-bdf70584d59f-kube-api-access-9gxz7\") pod \"payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d\" (UID: \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:46.100910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.100876 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:46.130402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.130372 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:46.294106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.293894 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:46.471300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.470883 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d"] Nov 02 14:09:46.483899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.483773 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs"] Nov 02 14:09:46.488339 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:09:46.488312 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752361dc_c22c_4225_9f6e_17f1034004a2.slice/crio-d3e206957ba8421d17c67333db0e597c6ecddccb2faa90d03d5b9889985d5a4c WatchSource:0}: Error finding container d3e206957ba8421d17c67333db0e597c6ecddccb2faa90d03d5b9889985d5a4c: Status 404 returned error can't find the container with id d3e206957ba8421d17c67333db0e597c6ecddccb2faa90d03d5b9889985d5a4c Nov 02 14:09:46.496707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.496679 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" event={"ID":"752361dc-c22c-4225-9f6e-17f1034004a2","Type":"ContainerStarted","Data":"d3e206957ba8421d17c67333db0e597c6ecddccb2faa90d03d5b9889985d5a4c"} Nov 02 14:09:46.498351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.498329 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" event={"ID":"f584b5b2-da4b-4a4e-be79-bdf70584d59f","Type":"ContainerStarted","Data":"19e67c0bdc6c52025bd0fca3e84cfca97723669fa7b73e2402050a7324bdcedd"} Nov 02 14:09:46.657726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.657688 5591 patch_prober.go:29] interesting pod/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:09:46.657922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:46.657754 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:09:47.506407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:47.506371 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" event={"ID":"f584b5b2-da4b-4a4e-be79-bdf70584d59f","Type":"ContainerStarted","Data":"57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300"} Nov 02 14:09:47.506923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:47.506437 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:47.507899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:47.507876 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" event={"ID":"752361dc-c22c-4225-9f6e-17f1034004a2","Type":"ContainerStarted","Data":"a36b431462e5f6047fc476f2c2c1d7dacf8e5cd01eac164f6dc1a917a05d1b30"} Nov 02 14:09:47.508025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:47.507925 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:47.553018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:47.549843 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" podStartSLOduration=2.5498240279999997 podStartE2EDuration="2.549824028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:47.549387011 +0000 UTC m=+1536.275008782" watchObservedRunningTime="2026-11-02 14:09:47.549824028 +0000 UTC m=+1536.275445800" Nov 02 14:09:47.594402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:47.594343 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" podStartSLOduration=2.594323501 podStartE2EDuration="2.594323501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:47.591230526 +0000 UTC m=+1536.316852297" watchObservedRunningTime="2026-11-02 14:09:47.594323501 +0000 UTC m=+1536.319945272" Nov 02 14:09:47.657453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:47.657416 5591 patch_prober.go:29] interesting pod/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:09:47.657632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:47.657473 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:09:48.472126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:48.470218 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:48.660877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:48.660843 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:09:48.662037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:48.662016 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:09:50.165628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:50.164487 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:51.872012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:51.870659 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:53.602090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:53.596411 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:53.774710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:53.774428 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:54.965299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:54.965262 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:55.274525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:55.274441 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:55.486006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:55.479893 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:55.671332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:55.671289 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:56.502045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:56.502012 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:09:57.054597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:57.054559 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:58.523198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:58.523164 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:09:58.523794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:58.523773 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:09:59.040967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:59.040911 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:09:59.830921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:09:59.830884 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:00.010441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:00.010405 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:00.852822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:00.852779 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:01.084635 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:01.084596 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:10:01.538394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:01.538356 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:01.707881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:01.707844 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:02.035449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.035410 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b"] Nov 02 14:10:02.042845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.042133 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:02.042845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.042139 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b"] Nov 02 14:10:02.057774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.057733 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-732d3a\"" Nov 02 14:10:02.070458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.070428 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-732d3a-dockercfg-g4xrl\"" Nov 02 14:10:02.142624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.142579 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxh8\" (UniqueName: \"kubernetes.io/projected/0aec9a81-5460-4411-a093-68ac8e84b56d-kube-api-access-hxxh8\") pod \"payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b\" (UID: \"0aec9a81-5460-4411-a093-68ac8e84b56d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:02.142624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.142628 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aec9a81-5460-4411-a093-68ac8e84b56d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b\" (UID: \"0aec9a81-5460-4411-a093-68ac8e84b56d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:02.244342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.244304 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxh8\" (UniqueName: \"kubernetes.io/projected/0aec9a81-5460-4411-a093-68ac8e84b56d-kube-api-access-hxxh8\") pod \"payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b\" (UID: \"0aec9a81-5460-4411-a093-68ac8e84b56d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:02.244540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.244352 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aec9a81-5460-4411-a093-68ac8e84b56d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b\" (UID: \"0aec9a81-5460-4411-a093-68ac8e84b56d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:02.245100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.245080 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aec9a81-5460-4411-a093-68ac8e84b56d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b\" (UID: \"0aec9a81-5460-4411-a093-68ac8e84b56d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:02.273188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.273145 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxh8\" (UniqueName: \"kubernetes.io/projected/0aec9a81-5460-4411-a093-68ac8e84b56d-kube-api-access-hxxh8\") pod \"payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b\" (UID: \"0aec9a81-5460-4411-a093-68ac8e84b56d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:02.371065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.363847 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:02.818586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.818553 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:02.866636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:02.866607 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b"] Nov 02 14:10:02.867540 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:10:02.867514 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aec9a81_5460_4411_a093_68ac8e84b56d.slice/crio-0e6542edd8a6dad9d6bdc4b8b7723669e261566ddc4bcfe4815906cb80d4d197 WatchSource:0}: Error finding container 0e6542edd8a6dad9d6bdc4b8b7723669e261566ddc4bcfe4815906cb80d4d197: Status 404 returned error can't find the container with id 0e6542edd8a6dad9d6bdc4b8b7723669e261566ddc4bcfe4815906cb80d4d197 Nov 02 14:10:03.832833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:03.832793 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" event={"ID":"0aec9a81-5460-4411-a093-68ac8e84b56d","Type":"ContainerStarted","Data":"fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060"} Nov 02 14:10:03.833355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:03.832861 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" event={"ID":"0aec9a81-5460-4411-a093-68ac8e84b56d","Type":"ContainerStarted","Data":"0e6542edd8a6dad9d6bdc4b8b7723669e261566ddc4bcfe4815906cb80d4d197"} Nov 02 14:10:03.833355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:03.832887 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:03.881130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:03.881068 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" podStartSLOduration=2.881050041 podStartE2EDuration="2.881050041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:03.876348386 +0000 UTC m=+1552.601970157" watchObservedRunningTime="2026-11-02 14:10:03.881050041 +0000 UTC m=+1552.606671814" Nov 02 14:10:05.083272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:05.083230 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:06.713633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:06.713580 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:06.856298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:06.856263 5591 generic.go:356] "Generic (PLEG): container finished" podID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerID="6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd" exitCode=0 Nov 02 14:10:06.856476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:06.856332 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" event={"ID":"82deb4cb-c2b7-4454-82d4-51a9f3959707","Type":"ContainerDied","Data":"6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd"} Nov 02 14:10:06.856841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:06.856820 5591 scope.go:122] "RemoveContainer" containerID="6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd" Nov 02 14:10:06.858491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:06.858459 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:10:06.858569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:06.858534 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:10:06.858615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:06.858566 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:10:06.894056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:06.893889 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:07.066114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:07.065490 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:07.685387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:07.685352 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:07.882209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:07.877149 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" event={"ID":"82deb4cb-c2b7-4454-82d4-51a9f3959707","Type":"ContainerStarted","Data":"2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1"} Nov 02 14:10:08.057636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:08.057367 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:10:08.059209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:08.059182 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 14:10:08.059371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:08.059237 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 14:10:08.117050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:08.117018 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:10:08.767552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:08.767469 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:09.056216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:09.056120 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 14:10:09.056698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:09.056216 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 14:10:10.052008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:10.044400 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 14:10:10.052008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:10.044484 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 14:10:10.206573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:10.206536 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:10:10.410127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:10.410083 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:10.584007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:10.580972 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:10.757716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:10.757615 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:11.050063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:11.047111 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 14:10:11.050063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:11.047177 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 14:10:12.054267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:12.054229 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 14:10:12.055676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:12.055637 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 14:10:12.289659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:12.289604 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:12.485751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:12.485718 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:13.044264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:13.044217 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 14:10:13.044489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:13.044284 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 14:10:14.036535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:14.036505 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:10:14.038593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:14.038563 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:10:14.561618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:14.561582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:14.843446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:14.843355 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:10:16.900976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:16.900878 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:18.586228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:18.586164 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:18.784216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:18.784182 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:19.241066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:19.240943 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:20.095290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.095256 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn"] Nov 02 14:10:20.095827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.095483 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" podUID="b8a9f158-2322-41bd-a21c-dd994ad10e74" containerName="maas-api" containerID="cri-o://c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b" gracePeriod=30 Nov 02 14:10:20.192588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.192539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:20.699799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.699764 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:10:20.793955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.793918 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-metrics-certs\") pod \"b8a9f158-2322-41bd-a21c-dd994ad10e74\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " Nov 02 14:10:20.794191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.794016 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a9f158-2322-41bd-a21c-dd994ad10e74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a9f158-2322-41bd-a21c-dd994ad10e74-openshift-service-ca\") pod \"b8a9f158-2322-41bd-a21c-dd994ad10e74\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " Nov 02 14:10:20.794191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.794056 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a9f158-2322-41bd-a21c-dd994ad10e74-kube-api-access-chg4z\" (UniqueName: \"kubernetes.io/projected/b8a9f158-2322-41bd-a21c-dd994ad10e74-kube-api-access-chg4z\") pod \"b8a9f158-2322-41bd-a21c-dd994ad10e74\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " Nov 02 14:10:20.794191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.794125 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls\") pod \"b8a9f158-2322-41bd-a21c-dd994ad10e74\" (UID: \"b8a9f158-2322-41bd-a21c-dd994ad10e74\") " Nov 02 14:10:20.794439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.794408 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a9f158-2322-41bd-a21c-dd994ad10e74-openshift-service-ca" pod "b8a9f158-2322-41bd-a21c-dd994ad10e74" (UID: "b8a9f158-2322-41bd-a21c-dd994ad10e74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:20.794875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.794845 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a9f158-2322-41bd-a21c-dd994ad10e74-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:20.798146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.798122 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-metrics-certs" pod "b8a9f158-2322-41bd-a21c-dd994ad10e74" (UID: "b8a9f158-2322-41bd-a21c-dd994ad10e74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:10:20.798239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.798157 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls" pod "b8a9f158-2322-41bd-a21c-dd994ad10e74" (UID: "b8a9f158-2322-41bd-a21c-dd994ad10e74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:10:20.798239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.798177 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a9f158-2322-41bd-a21c-dd994ad10e74-kube-api-access-chg4z" pod "b8a9f158-2322-41bd-a21c-dd994ad10e74" (UID: "b8a9f158-2322-41bd-a21c-dd994ad10e74"). InnerVolumeSpecName "kube-api-access-chg4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:20.908090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.900500 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:20.908090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.900570 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chg4z\" (UniqueName: \"kubernetes.io/projected/b8a9f158-2322-41bd-a21c-dd994ad10e74-kube-api-access-chg4z\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:20.908090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.900625 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8a9f158-2322-41bd-a21c-dd994ad10e74-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:20.957675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.957095 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:20.987957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.987924 5591 generic.go:356] "Generic (PLEG): container finished" podID="b8a9f158-2322-41bd-a21c-dd994ad10e74" containerID="c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b" exitCode=0 Nov 02 14:10:20.988230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.987975 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" Nov 02 14:10:20.988230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.988010 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" event={"ID":"b8a9f158-2322-41bd-a21c-dd994ad10e74","Type":"ContainerDied","Data":"c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b"} Nov 02 14:10:20.988230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.988087 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn" event={"ID":"b8a9f158-2322-41bd-a21c-dd994ad10e74","Type":"ContainerDied","Data":"d77abf630d83a4c17080bbfa8678653aea9fbbe13bc43a13357e41db1e5129b0"} Nov 02 14:10:20.988230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.988101 5591 scope.go:122] "RemoveContainer" containerID="c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b" Nov 02 14:10:20.999016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.998996 5591 scope.go:122] "RemoveContainer" containerID="c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b" Nov 02 14:10:20.999287 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:20.999266 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b\": container with ID starting with c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b not found: ID does not exist" containerID="c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b" Nov 02 14:10:20.999344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:20.999295 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b"} err="failed to get container status \"c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b\": rpc error: code = NotFound desc = could not find container \"c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b\": container with ID starting with c9a2e6f3b1c988ae7e550cfc4ae9d75c9b5270598915364645aea756c7775d1b not found: ID does not exist" Nov 02 14:10:21.047233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:21.047198 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn"] Nov 02 14:10:21.060189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:21.060160 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b7d400-5b477cfc-sfdqn"] Nov 02 14:10:21.433574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:21.433540 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a9f158-2322-41bd-a21c-dd994ad10e74" path="/var/lib/kubelet/pods/b8a9f158-2322-41bd-a21c-dd994ad10e74/volumes" Nov 02 14:10:22.579441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:22.579402 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:22.873212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:22.873123 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:24.489912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:24.489874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:25.181246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:25.181077 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:25.349954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:25.349900 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:25.999692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:25.999653 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:26.367447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:26.367314 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:26.546087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:26.546048 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:28.297514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:28.297473 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:29.957718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:29.957683 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr"] Nov 02 14:10:29.958251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:29.957944 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" containerID="cri-o://6e7c7485172de18089267f9e00153915b3527cba3449021dbbfa794fd50d21b1" gracePeriod=30 Nov 02 14:10:30.093004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:30.092892 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:30.273632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:30.273600 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6"] Nov 02 14:10:30.273826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:30.273810 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" containerID="cri-o://634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d" gracePeriod=30 Nov 02 14:10:31.094280 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:31.094231 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:10:31.892520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:31.892479 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:33.599220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:33.599187 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5"] Nov 02 14:10:33.599685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:33.599417 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" containerID="cri-o://c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17" gracePeriod=30 Nov 02 14:10:34.139157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:34.139118 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:34.143856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:34.143826 5591 generic.go:356] "Generic (PLEG): container finished" podID="92939b14-a553-47be-9bd8-a5de527a88ca" containerID="030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e" exitCode=0 Nov 02 14:10:34.144012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:34.143872 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" event={"ID":"92939b14-a553-47be-9bd8-a5de527a88ca","Type":"ContainerDied","Data":"030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e"} Nov 02 14:10:34.144353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:34.144335 5591 scope.go:122] "RemoveContainer" containerID="030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e" Nov 02 14:10:34.146071 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:34.146031 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:10:34.146193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:34.146130 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:10:34.146193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:34.146183 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:10:35.056011 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:35.031740 5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34932ab0_cb39_43e2_b9e1_ce8cb25b5e57.slice/crio-6e7c7485172de18089267f9e00153915b3527cba3449021dbbfa794fd50d21b1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34932ab0_cb39_43e2_b9e1_ce8cb25b5e57.slice/crio-conmon-6e7c7485172de18089267f9e00153915b3527cba3449021dbbfa794fd50d21b1.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:10:35.174479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.174444 5591 generic.go:356] "Generic (PLEG): container finished" podID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerID="6e7c7485172de18089267f9e00153915b3527cba3449021dbbfa794fd50d21b1" exitCode=0 Nov 02 14:10:35.174686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.174544 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" event={"ID":"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57","Type":"ContainerDied","Data":"6e7c7485172de18089267f9e00153915b3527cba3449021dbbfa794fd50d21b1"} Nov 02 14:10:35.174686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.174589 5591 scope.go:122] "RemoveContainer" containerID="4099ca9e33f9c33e61dcc098d110a5494783071df09f4a113933c8aeeb589219" Nov 02 14:10:35.195499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.195417 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" event={"ID":"92939b14-a553-47be-9bd8-a5de527a88ca","Type":"ContainerStarted","Data":"b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740"} Nov 02 14:10:35.288256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.288228 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 14:10:35.395462 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395429 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-envoy\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.395635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395482 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-data\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.395635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395528 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-certs\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.395756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395669 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-socket\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.395756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395708 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-kube-api-access-mhjqd\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-kube-api-access-mhjqd\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.395756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395739 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-token\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-token\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.395920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395764 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-credential-socket\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.395920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395897 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istiod-ca-cert\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.396074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.395935 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-podinfo\") pod \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\" (UID: \"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57\") " Nov 02 14:10:35.396074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.396008 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-certs" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.396464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.396436 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-credential-socket" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.396698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.396675 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.396798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.396702 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.397046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.397021 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-socket" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.397458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.397426 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-data" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.397737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.397716 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istiod-ca-cert" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:35.399332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.398839 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-podinfo" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:10:35.402534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.402500 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-envoy" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.402630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.402552 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-token" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:35.404189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.404166 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-kube-api-access-mhjqd" pod "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" (UID: "34932ab0-cb39-43e2-b9e1-ce8cb25b5e57"). InnerVolumeSpecName "kube-api-access-mhjqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:35.497936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.497902 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.498101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.497942 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.498101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.497957 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.498101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.497970 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.498101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.497999 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.498101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.498013 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhjqd\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-kube-api-access-mhjqd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.498101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.498025 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.581393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.581367 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 14:10:35.700742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.700645 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-credential-socket\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.700742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.700723 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06f128ca-a03d-471b-a951-4d19cddd1785-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f128ca-a03d-471b-a951-4d19cddd1785-istio-podinfo\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.701056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.700826 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-certs\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.701056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.700859 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-envoy\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.701056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.700889 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-istio-token\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-istio-token\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.701056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.700935 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-socket\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.701303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.701064 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-kube-api-access-4p82m\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-kube-api-access-4p82m\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.701303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.701166 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f128ca-a03d-471b-a951-4d19cddd1785-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f128ca-a03d-471b-a951-4d19cddd1785-istiod-ca-cert\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.701303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.701253 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-credential-socket" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.701477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.701306 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-data\") pod \"06f128ca-a03d-471b-a951-4d19cddd1785\" (UID: \"06f128ca-a03d-471b-a951-4d19cddd1785\") " Nov 02 14:10:35.701477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.701454 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-socket" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.701670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.701641 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-data" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.701796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.701697 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f128ca-a03d-471b-a951-4d19cddd1785-istiod-ca-cert" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:35.701882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.701822 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-certs" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.702152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.702126 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f128ca-a03d-471b-a951-4d19cddd1785-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.702250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.702156 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.702250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.702171 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.702250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.702188 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.702250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.702201 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.703449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.703423 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06f128ca-a03d-471b-a951-4d19cddd1785-istio-podinfo" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:10:35.703553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.703505 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-istio-token" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:35.703553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.703513 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-envoy" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:35.703553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.703530 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-kube-api-access-4p82m" pod "06f128ca-a03d-471b-a951-4d19cddd1785" (UID: "06f128ca-a03d-471b-a951-4d19cddd1785"). InnerVolumeSpecName "kube-api-access-4p82m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:35.771929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.771891 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:10:35.772183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.772155 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 14:10:35.772270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.772228 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 14:10:35.813017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.804422 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p82m\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-kube-api-access-4p82m\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.813017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.804460 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f128ca-a03d-471b-a951-4d19cddd1785-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.813017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.804476 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f128ca-a03d-471b-a951-4d19cddd1785-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.813017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.804488 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f128ca-a03d-471b-a951-4d19cddd1785-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:35.859063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:35.858746 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:36.203883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.203853 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" Nov 02 14:10:36.204352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.203858 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr" event={"ID":"34932ab0-cb39-43e2-b9e1-ce8cb25b5e57","Type":"ContainerDied","Data":"6ffab15442717c10900702b471c9ca4c0d7d4b4091648b402f0079d45fa8a51c"} Nov 02 14:10:36.204352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.204026 5591 scope.go:122] "RemoveContainer" containerID="6e7c7485172de18089267f9e00153915b3527cba3449021dbbfa794fd50d21b1" Nov 02 14:10:36.205361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.205337 5591 generic.go:356] "Generic (PLEG): container finished" podID="06f128ca-a03d-471b-a951-4d19cddd1785" containerID="634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d" exitCode=0 Nov 02 14:10:36.205447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.205373 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" event={"ID":"06f128ca-a03d-471b-a951-4d19cddd1785","Type":"ContainerDied","Data":"634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d"} Nov 02 14:10:36.205447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.205403 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" Nov 02 14:10:36.205447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.205433 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6" event={"ID":"06f128ca-a03d-471b-a951-4d19cddd1785","Type":"ContainerDied","Data":"84ed00777f8bbd9254371eddf06d7e66fd2097e940c6d1ee1ac4c6f6d6375f7c"} Nov 02 14:10:36.215236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.215219 5591 scope.go:122] "RemoveContainer" containerID="634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d" Nov 02 14:10:36.224369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.224351 5591 scope.go:122] "RemoveContainer" containerID="634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d" Nov 02 14:10:36.224615 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:36.224595 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d\": container with ID starting with 634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d not found: ID does not exist" containerID="634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d" Nov 02 14:10:36.224669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.224627 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d"} err="failed to get container status \"634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d\": rpc error: code = NotFound desc = could not find container \"634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d\": container with ID starting with 634a170ce7531c0da922bb501e73539f4d5183becd0ab1087e9ad002b58b9e2d not found: ID does not exist" Nov 02 14:10:36.270676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.270637 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr"] Nov 02 14:10:36.283857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.283824 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dhbtr"] Nov 02 14:10:36.292566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.292542 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24"] Nov 02 14:10:36.292767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.292746 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" containerID="cri-o://4c10841914bc015c30005f36f9ad33e58c234423389d1498718c157aacbdb20e" gracePeriod=30 Nov 02 14:10:36.327485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.327443 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6"] Nov 02 14:10:36.350010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.331579 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-hzzd6"] Nov 02 14:10:36.771693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.771657 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 14:10:36.771861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:36.771710 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 14:10:37.433898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:37.433856 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" path="/var/lib/kubelet/pods/06f128ca-a03d-471b-a951-4d19cddd1785/volumes" Nov 02 14:10:37.435730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:37.435696 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" path="/var/lib/kubelet/pods/34932ab0-cb39-43e2-b9e1-ce8cb25b5e57/volumes" Nov 02 14:10:37.490041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:37.489523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:37.772142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:37.772057 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 14:10:37.772142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:37.772122 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 14:10:38.456869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:38.456823 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:38.771723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:38.771692 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 14:10:38.771929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:38.771903 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 14:10:38.979445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:38.979416 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.055551 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-envoy\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.055707 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-data\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.055745 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-token\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.055788 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-credential-socket\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.055853 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-kube-api-access-lxlx9\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-kube-api-access-lxlx9\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.055906 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istiod-ca-cert\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.055978 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-podinfo\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.056173 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-certs\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.058254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.056225 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-socket\") pod \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\" (UID: \"8d8b9c04-2f85-4bed-96ce-7ea07e69317d\") " Nov 02 14:10:39.062282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.061305 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-certs" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:39.062282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.061488 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-credential-socket" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:39.062282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.061679 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-data" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:39.063711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.063687 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istiod-ca-cert" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:39.064088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.064065 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-socket" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:39.084937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.083430 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-envoy" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:39.084937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.084894 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-kube-api-access-lxlx9" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "kube-api-access-lxlx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:39.085150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.084935 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-token" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:39.095350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.095306 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-podinfo" pod "8d8b9c04-2f85-4bed-96ce-7ea07e69317d" (UID: "8d8b9c04-2f85-4bed-96ce-7ea07e69317d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166424 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlx9\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-kube-api-access-lxlx9\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166471 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166487 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166500 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166514 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166528 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166542 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166555 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.166727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.166568 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b9c04-2f85-4bed-96ce-7ea07e69317d-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:39.375823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.373818 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:39.437014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.434512 5591 generic.go:356] "Generic (PLEG): container finished" podID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerID="c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17" exitCode=0 Nov 02 14:10:39.457038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.438714 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" Nov 02 14:10:39.484274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.484040 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" event={"ID":"8d8b9c04-2f85-4bed-96ce-7ea07e69317d","Type":"ContainerDied","Data":"c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17"} Nov 02 14:10:39.484274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.484075 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5" event={"ID":"8d8b9c04-2f85-4bed-96ce-7ea07e69317d","Type":"ContainerDied","Data":"8f1c9cd451ee2279e0cbedbc704ea9e87817d29a57ea92bb16abaf85d052b052"} Nov 02 14:10:39.484274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.484091 5591 scope.go:122] "RemoveContainer" containerID="c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17" Nov 02 14:10:39.498931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.498908 5591 scope.go:122] "RemoveContainer" containerID="c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17" Nov 02 14:10:39.499298 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:39.499269 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17\": container with ID starting with c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17 not found: ID does not exist" containerID="c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17" Nov 02 14:10:39.499385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.499308 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17"} err="failed to get container status \"c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17\": rpc error: code = NotFound desc = could not find container \"c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17\": container with ID starting with c0fc6e5635edd5444b828658375c8288d481013f421bf4a6a7856b9bb528fa17 not found: ID does not exist" Nov 02 14:10:39.520581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.520557 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5"] Nov 02 14:10:39.524782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.524756 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-nvjl5"] Nov 02 14:10:39.772128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.772093 5591 patch_prober.go:29] interesting pod/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 14:10:39.772308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:39.772148 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 14:10:40.776738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:40.776700 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:10:40.777278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:40.776978 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:10:40.777788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:40.777770 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:10:41.335007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.334713 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:41.435121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.435081 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" path="/var/lib/kubelet/pods/8d8b9c04-2f85-4bed-96ce-7ea07e69317d/volumes" Nov 02 14:10:41.455477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.455439 5591 generic.go:356] "Generic (PLEG): container finished" podID="14c18551-b05a-4964-9964-dd5d7a79e943" containerID="4c10841914bc015c30005f36f9ad33e58c234423389d1498718c157aacbdb20e" exitCode=0 Nov 02 14:10:41.455636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.455512 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" event={"ID":"14c18551-b05a-4964-9964-dd5d7a79e943","Type":"ContainerDied","Data":"4c10841914bc015c30005f36f9ad33e58c234423389d1498718c157aacbdb20e"} Nov 02 14:10:41.575864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.575842 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 14:10:41.713884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.713852 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c18551-b05a-4964-9964-dd5d7a79e943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c18551-b05a-4964-9964-dd5d7a79e943-istiod-ca-cert\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714028 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-envoy\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714068 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14c18551-b05a-4964-9964-dd5d7a79e943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c18551-b05a-4964-9964-dd5d7a79e943-istio-podinfo\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714117 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-data\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714143 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-istio-token\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-istio-token\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714231 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-kube-api-access-rnfph\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-kube-api-access-rnfph\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714280 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c18551-b05a-4964-9964-dd5d7a79e943-istiod-ca-cert" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:41.714347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714306 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-certs\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714349 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-credential-socket\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714393 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-socket\") pod \"14c18551-b05a-4964-9964-dd5d7a79e943\" (UID: \"14c18551-b05a-4964-9964-dd5d7a79e943\") " Nov 02 14:10:41.714503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714452 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-data" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:41.714682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714660 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-credential-socket" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:41.714778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.714754 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-socket" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:41.715100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.715073 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:41.715252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.715103 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:41.715252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.715122 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c18551-b05a-4964-9964-dd5d7a79e943-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:41.715252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.715136 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:41.715252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.715182 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-certs" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:41.716459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.716434 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14c18551-b05a-4964-9964-dd5d7a79e943-istio-podinfo" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:10:41.716559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.716457 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-istio-token" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:41.716559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.716443 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-envoy" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:41.716559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.716515 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-kube-api-access-rnfph" pod "14c18551-b05a-4964-9964-dd5d7a79e943" (UID: "14c18551-b05a-4964-9964-dd5d7a79e943"). InnerVolumeSpecName "kube-api-access-rnfph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:41.816401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.816367 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:41.816401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.816398 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c18551-b05a-4964-9964-dd5d7a79e943-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:41.816814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.816409 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:41.816814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.816423 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnfph\" (UniqueName: \"kubernetes.io/projected/14c18551-b05a-4964-9964-dd5d7a79e943-kube-api-access-rnfph\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:41.816814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:41.816435 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c18551-b05a-4964-9964-dd5d7a79e943-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:42.463723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:42.463688 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" Nov 02 14:10:42.463723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:42.463710 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24" event={"ID":"14c18551-b05a-4964-9964-dd5d7a79e943","Type":"ContainerDied","Data":"22fe963ac7efbb0d2c74d941fe4bbc776547ec4ab6ad0191c8dc9c44195478f7"} Nov 02 14:10:42.464023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:42.463775 5591 scope.go:122] "RemoveContainer" containerID="4c10841914bc015c30005f36f9ad33e58c234423389d1498718c157aacbdb20e" Nov 02 14:10:42.531802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:42.531773 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24"] Nov 02 14:10:42.544784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:42.544755 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-czk24"] Nov 02 14:10:42.807486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:42.807385 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:43.014024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:43.013964 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:43.190320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:43.190280 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:43.432122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:43.432086 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" path="/var/lib/kubelet/pods/14c18551-b05a-4964-9964-dd5d7a79e943/volumes" Nov 02 14:10:43.576587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:43.576493 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:44.026067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:44.026036 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d"] Nov 02 14:10:44.066499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:44.066463 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs"] Nov 02 14:10:44.081880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:44.081852 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:44.247867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:44.247829 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:44.445126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:44.438550 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:44.868331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:44.868228 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:45.743615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:45.743575 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:45.982282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:45.982250 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8"] Nov 02 14:10:45.982464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:45.982436 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" containerID="cri-o://14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb" gracePeriod=30 Nov 02 14:10:47.385005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:47.374632 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:48.605200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:48.605165 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:10:49.026893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.026827 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" podUID="f584b5b2-da4b-4a4e-be79-bdf70584d59f" containerName="payload-processing" containerID="cri-o://57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300" gracePeriod=25 Nov 02 14:10:49.067975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.067940 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" podUID="752361dc-c22c-4225-9f6e-17f1034004a2" containerName="payload-pre-processing" containerID="cri-o://a36b431462e5f6047fc476f2c2c1d7dacf8e5cd01eac164f6dc1a917a05d1b30" gracePeriod=25 Nov 02 14:10:49.348121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.345526 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:49.581697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.580297 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:10:49.583607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.583564 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f584b5b2-da4b-4a4e-be79-bdf70584d59f-kube-api-access-9gxz7\" (UniqueName: \"kubernetes.io/projected/f584b5b2-da4b-4a4e-be79-bdf70584d59f-kube-api-access-9gxz7\") pod \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\" (UID: \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\") " Nov 02 14:10:49.584332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.584291 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f584b5b2-da4b-4a4e-be79-bdf70584d59f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584b5b2-da4b-4a4e-be79-bdf70584d59f-plugins-config-volume\") pod \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\" (UID: \"f584b5b2-da4b-4a4e-be79-bdf70584d59f\") " Nov 02 14:10:49.593473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.593328 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f584b5b2-da4b-4a4e-be79-bdf70584d59f-plugins-config-volume" pod "f584b5b2-da4b-4a4e-be79-bdf70584d59f" (UID: "f584b5b2-da4b-4a4e-be79-bdf70584d59f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:49.627082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.604869 5591 generic.go:356] "Generic (PLEG): container finished" podID="752361dc-c22c-4225-9f6e-17f1034004a2" containerID="a36b431462e5f6047fc476f2c2c1d7dacf8e5cd01eac164f6dc1a917a05d1b30" exitCode=0 Nov 02 14:10:49.627082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.605156 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" event={"ID":"752361dc-c22c-4225-9f6e-17f1034004a2","Type":"ContainerDied","Data":"a36b431462e5f6047fc476f2c2c1d7dacf8e5cd01eac164f6dc1a917a05d1b30"} Nov 02 14:10:49.627082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.607282 5591 generic.go:356] "Generic (PLEG): container finished" podID="f584b5b2-da4b-4a4e-be79-bdf70584d59f" containerID="57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300" exitCode=0 Nov 02 14:10:49.627082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.607324 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" event={"ID":"f584b5b2-da4b-4a4e-be79-bdf70584d59f","Type":"ContainerDied","Data":"57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300"} Nov 02 14:10:49.627082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.607364 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" event={"ID":"f584b5b2-da4b-4a4e-be79-bdf70584d59f","Type":"ContainerDied","Data":"19e67c0bdc6c52025bd0fca3e84cfca97723669fa7b73e2402050a7324bdcedd"} Nov 02 14:10:49.627082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.607390 5591 scope.go:122] "RemoveContainer" containerID="57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300" Nov 02 14:10:49.627082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.607644 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d" Nov 02 14:10:49.628510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.628482 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f584b5b2-da4b-4a4e-be79-bdf70584d59f-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:49.721925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.720275 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f584b5b2-da4b-4a4e-be79-bdf70584d59f-kube-api-access-9gxz7" pod "f584b5b2-da4b-4a4e-be79-bdf70584d59f" (UID: "f584b5b2-da4b-4a4e-be79-bdf70584d59f"). InnerVolumeSpecName "kube-api-access-9gxz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:49.724613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.724439 5591 scope.go:122] "RemoveContainer" containerID="57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300" Nov 02 14:10:49.726476 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:49.726443 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300\": container with ID starting with 57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300 not found: ID does not exist" containerID="57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300" Nov 02 14:10:49.726631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.726491 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300"} err="failed to get container status \"57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300\": rpc error: code = NotFound desc = could not find container \"57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300\": container with ID starting with 57892865acf1a6b038dbe4280c3083a4b2e8e1136f8c74be7e3e5e0006d4a300 not found: ID does not exist" Nov 02 14:10:49.731034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.731009 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:10:49.740251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.740218 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gxz7\" (UniqueName: \"kubernetes.io/projected/f584b5b2-da4b-4a4e-be79-bdf70584d59f-kube-api-access-9gxz7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:49.841167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.841134 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752361dc-c22c-4225-9f6e-17f1034004a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752361dc-c22c-4225-9f6e-17f1034004a2-plugins-config-volume\") pod \"752361dc-c22c-4225-9f6e-17f1034004a2\" (UID: \"752361dc-c22c-4225-9f6e-17f1034004a2\") " Nov 02 14:10:49.841346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.841221 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752361dc-c22c-4225-9f6e-17f1034004a2-kube-api-access-j2k8j\" (UniqueName: \"kubernetes.io/projected/752361dc-c22c-4225-9f6e-17f1034004a2-kube-api-access-j2k8j\") pod \"752361dc-c22c-4225-9f6e-17f1034004a2\" (UID: \"752361dc-c22c-4225-9f6e-17f1034004a2\") " Nov 02 14:10:49.841526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.841499 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752361dc-c22c-4225-9f6e-17f1034004a2-plugins-config-volume" pod "752361dc-c22c-4225-9f6e-17f1034004a2" (UID: "752361dc-c22c-4225-9f6e-17f1034004a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:49.841626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.841613 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752361dc-c22c-4225-9f6e-17f1034004a2-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:49.843361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.843336 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752361dc-c22c-4225-9f6e-17f1034004a2-kube-api-access-j2k8j" pod "752361dc-c22c-4225-9f6e-17f1034004a2" (UID: "752361dc-c22c-4225-9f6e-17f1034004a2"). InnerVolumeSpecName "kube-api-access-j2k8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:49.943303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.943272 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2k8j\" (UniqueName: \"kubernetes.io/projected/752361dc-c22c-4225-9f6e-17f1034004a2-kube-api-access-j2k8j\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:49.957692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.957660 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d"] Nov 02 14:10:49.965500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:49.965474 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-5f87d899f7-9ql8d"] Nov 02 14:10:50.617118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:50.617092 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" Nov 02 14:10:50.617307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:50.617127 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs" event={"ID":"752361dc-c22c-4225-9f6e-17f1034004a2","Type":"ContainerDied","Data":"d3e206957ba8421d17c67333db0e597c6ecddccb2faa90d03d5b9889985d5a4c"} Nov 02 14:10:50.617307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:50.617188 5591 scope.go:122] "RemoveContainer" containerID="a36b431462e5f6047fc476f2c2c1d7dacf8e5cd01eac164f6dc1a917a05d1b30" Nov 02 14:10:50.671897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:50.671868 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs"] Nov 02 14:10:50.683688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:50.683660 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-858c964599-vnlgs"] Nov 02 14:10:50.996094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:50.992280 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:51.247848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.247786 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:10:51.271909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.271878 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:51.373311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373281 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-kube-api-access-zhb2r\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-kube-api-access-zhb2r\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.373514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373375 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-credential-socket\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.373514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373474 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-certs\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.373647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373565 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-data\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.373647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373586 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ef2f4d-98a0-41d0-8a68-75a21356effb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ef2f4d-98a0-41d0-8a68-75a21356effb-istiod-ca-cert\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.373647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373604 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-credential-socket" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:51.373809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373657 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-socket\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.373809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373729 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-certs" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:51.373809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373739 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-token\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-token\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.373809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373781 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-envoy\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.374069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373852 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-data" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:51.374069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373918 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-podinfo\") pod \"47ef2f4d-98a0-41d0-8a68-75a21356effb\" (UID: \"47ef2f4d-98a0-41d0-8a68-75a21356effb\") " Nov 02 14:10:51.374069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373928 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-socket" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:51.374069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.373961 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ef2f4d-98a0-41d0-8a68-75a21356effb-istiod-ca-cert" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:51.374552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.374519 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.374552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.374546 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.374713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.374563 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.374713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.374577 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ef2f4d-98a0-41d0-8a68-75a21356effb-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.374713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.374590 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.375942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.375919 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-kube-api-access-zhb2r" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "kube-api-access-zhb2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:51.376077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.375976 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-token" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:51.376077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.376039 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-envoy" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:10:51.376370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.376351 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-podinfo" pod "47ef2f4d-98a0-41d0-8a68-75a21356effb" (UID: "47ef2f4d-98a0-41d0-8a68-75a21356effb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:10:51.437337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.437295 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752361dc-c22c-4225-9f6e-17f1034004a2" path="/var/lib/kubelet/pods/752361dc-c22c-4225-9f6e-17f1034004a2/volumes" Nov 02 14:10:51.438238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.438208 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f584b5b2-da4b-4a4e-be79-bdf70584d59f" path="/var/lib/kubelet/pods/f584b5b2-da4b-4a4e-be79-bdf70584d59f/volumes" Nov 02 14:10:51.476223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.476191 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.476223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.476219 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.476223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.476229 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ef2f4d-98a0-41d0-8a68-75a21356effb-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.476430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.476239 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhb2r\" (UniqueName: \"kubernetes.io/projected/47ef2f4d-98a0-41d0-8a68-75a21356effb-kube-api-access-zhb2r\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:51.625466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.625383 5591 generic.go:356] "Generic (PLEG): container finished" podID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerID="14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb" exitCode=0 Nov 02 14:10:51.625466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.625439 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" Nov 02 14:10:51.625661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.625465 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" event={"ID":"47ef2f4d-98a0-41d0-8a68-75a21356effb","Type":"ContainerDied","Data":"14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb"} Nov 02 14:10:51.625661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.625519 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8" event={"ID":"47ef2f4d-98a0-41d0-8a68-75a21356effb","Type":"ContainerDied","Data":"4ff255ca697a932fca76b5c91299d70177b505fcd917bd8ffe701ce983d1eb86"} Nov 02 14:10:51.625661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.625543 5591 scope.go:122] "RemoveContainer" containerID="14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb" Nov 02 14:10:51.635434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.635402 5591 scope.go:122] "RemoveContainer" containerID="14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb" Nov 02 14:10:51.635724 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:51.635702 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb\": container with ID starting with 14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb not found: ID does not exist" containerID="14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb" Nov 02 14:10:51.635811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.635733 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb"} err="failed to get container status \"14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb\": rpc error: code = NotFound desc = could not find container \"14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb\": container with ID starting with 14df71d6eddf5d4edc16660224dd7fb32823a8bd5da9f30ce4ee50211d2ed0cb not found: ID does not exist" Nov 02 14:10:51.691195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.691164 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8"] Nov 02 14:10:51.705822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.705791 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-c6qk8"] Nov 02 14:10:51.981340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:51.981306 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:52.142468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:52.142426 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:52.575295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:52.575258 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j"] Nov 02 14:10:52.610185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:52.610152 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2"] Nov 02 14:10:52.764479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:52.764447 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:52.944971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:52.944928 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:53.107244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:53.107195 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:53.434059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:53.434028 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" path="/var/lib/kubelet/pods/47ef2f4d-98a0-41d0-8a68-75a21356effb/volumes" Nov 02 14:10:54.677020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:54.675898 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:56.494108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:56.494078 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-t792x_566435d0-5ce9-4ef4-9a61-9ebb69b8d17d/payload-processing/0.log" Nov 02 14:10:56.611098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:56.610465 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:56.737131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:56.737104 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-t792x_566435d0-5ce9-4ef4-9a61-9ebb69b8d17d/payload-processing/0.log" Nov 02 14:10:56.939677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:56.939645 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b_122ac438-de00-431f-829b-a2d925799aa2/payload-processing/0.log" Nov 02 14:10:57.141440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.141408 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b_122ac438-de00-431f-829b-a2d925799aa2/payload-processing/0.log" Nov 02 14:10:57.232034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.231916 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:57.576106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.576022 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" podUID="f3da4df8-c04e-42aa-bb51-df99e073b242" containerName="payload-processing" containerID="cri-o://2e22581b56688628061f40bf7b864c21975c700763186386654515b2819cdd2c" gracePeriod=25 Nov 02 14:10:57.611358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.611301 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" podUID="6d30ac92-1b49-41e0-90fa-fe108d5679b8" containerName="payload-pre-processing" containerID="cri-o://95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440" gracePeriod=25 Nov 02 14:10:57.709301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.709127 5591 generic.go:356] "Generic (PLEG): container finished" podID="f3da4df8-c04e-42aa-bb51-df99e073b242" containerID="2e22581b56688628061f40bf7b864c21975c700763186386654515b2819cdd2c" exitCode=0 Nov 02 14:10:57.709301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.709238 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" event={"ID":"f3da4df8-c04e-42aa-bb51-df99e073b242","Type":"ContainerDied","Data":"2e22581b56688628061f40bf7b864c21975c700763186386654515b2819cdd2c"} Nov 02 14:10:57.887850 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.887821 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:10:57.920672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.920645 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:10:57.923795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.923773 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3da4df8-c04e-42aa-bb51-df99e073b242-kube-api-access-rwpxm\" (UniqueName: \"kubernetes.io/projected/f3da4df8-c04e-42aa-bb51-df99e073b242-kube-api-access-rwpxm\") pod \"f3da4df8-c04e-42aa-bb51-df99e073b242\" (UID: \"f3da4df8-c04e-42aa-bb51-df99e073b242\") " Nov 02 14:10:57.923926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.923901 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d30ac92-1b49-41e0-90fa-fe108d5679b8-kube-api-access-rmlpv\" (UniqueName: \"kubernetes.io/projected/6d30ac92-1b49-41e0-90fa-fe108d5679b8-kube-api-access-rmlpv\") pod \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\" (UID: \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\") " Nov 02 14:10:57.924029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.923967 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d30ac92-1b49-41e0-90fa-fe108d5679b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d30ac92-1b49-41e0-90fa-fe108d5679b8-plugins-config-volume\") pod \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\" (UID: \"6d30ac92-1b49-41e0-90fa-fe108d5679b8\") " Nov 02 14:10:57.924098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.924079 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3da4df8-c04e-42aa-bb51-df99e073b242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3da4df8-c04e-42aa-bb51-df99e073b242-plugins-config-volume\") pod \"f3da4df8-c04e-42aa-bb51-df99e073b242\" (UID: \"f3da4df8-c04e-42aa-bb51-df99e073b242\") " Nov 02 14:10:57.924472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.924443 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d30ac92-1b49-41e0-90fa-fe108d5679b8-plugins-config-volume" pod "6d30ac92-1b49-41e0-90fa-fe108d5679b8" (UID: "6d30ac92-1b49-41e0-90fa-fe108d5679b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:57.924472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.924462 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3da4df8-c04e-42aa-bb51-df99e073b242-plugins-config-volume" pod "f3da4df8-c04e-42aa-bb51-df99e073b242" (UID: "f3da4df8-c04e-42aa-bb51-df99e073b242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:57.924649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.924614 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d30ac92-1b49-41e0-90fa-fe108d5679b8-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:57.924649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.924632 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3da4df8-c04e-42aa-bb51-df99e073b242-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:57.925993 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.925961 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3da4df8-c04e-42aa-bb51-df99e073b242-kube-api-access-rwpxm" pod "f3da4df8-c04e-42aa-bb51-df99e073b242" (UID: "f3da4df8-c04e-42aa-bb51-df99e073b242"). InnerVolumeSpecName "kube-api-access-rwpxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:57.926274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:57.926253 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d30ac92-1b49-41e0-90fa-fe108d5679b8-kube-api-access-rmlpv" pod "6d30ac92-1b49-41e0-90fa-fe108d5679b8" (UID: "6d30ac92-1b49-41e0-90fa-fe108d5679b8"). InnerVolumeSpecName "kube-api-access-rmlpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:58.026077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.026036 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmlpv\" (UniqueName: \"kubernetes.io/projected/6d30ac92-1b49-41e0-90fa-fe108d5679b8-kube-api-access-rmlpv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:58.026077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.026075 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwpxm\" (UniqueName: \"kubernetes.io/projected/f3da4df8-c04e-42aa-bb51-df99e073b242-kube-api-access-rwpxm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:10:58.265945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.265908 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:58.497165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.493297 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:58.733544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.733511 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" event={"ID":"f3da4df8-c04e-42aa-bb51-df99e073b242","Type":"ContainerDied","Data":"f67a9be0ef68d11a1b9239c7788bf8de279e2d8107911c0b7aa83b979d7b86da"} Nov 02 14:10:58.734112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.734092 5591 scope.go:122] "RemoveContainer" containerID="2e22581b56688628061f40bf7b864c21975c700763186386654515b2819cdd2c" Nov 02 14:10:58.734377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.734361 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j" Nov 02 14:10:58.754072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.754037 5591 generic.go:356] "Generic (PLEG): container finished" podID="6d30ac92-1b49-41e0-90fa-fe108d5679b8" containerID="95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440" exitCode=0 Nov 02 14:10:58.754339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.754313 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" event={"ID":"6d30ac92-1b49-41e0-90fa-fe108d5679b8","Type":"ContainerDied","Data":"95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440"} Nov 02 14:10:58.754499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.754480 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" event={"ID":"6d30ac92-1b49-41e0-90fa-fe108d5679b8","Type":"ContainerDied","Data":"694b948aa68e0de0cf67c38050142e5884974ed3d73cdbcbf272faf7446f5830"} Nov 02 14:10:58.754765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.754747 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2" Nov 02 14:10:58.773694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.773667 5591 scope.go:122] "RemoveContainer" containerID="95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440" Nov 02 14:10:58.784937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.784913 5591 scope.go:122] "RemoveContainer" containerID="95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440" Nov 02 14:10:58.785281 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:10:58.785254 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440\": container with ID starting with 95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440 not found: ID does not exist" containerID="95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440" Nov 02 14:10:58.785358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.785291 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440"} err="failed to get container status \"95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440\": rpc error: code = NotFound desc = could not find container \"95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440\": container with ID starting with 95f6c867eff70d4eb54491659027f58b6ba6b8c6516a11c7e0bb26b31c3ea440 not found: ID does not exist" Nov 02 14:10:58.822704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.822675 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j"] Nov 02 14:10:58.837019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.836972 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cc8d39-76d999c875-5rd6j"] Nov 02 14:10:58.869776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.869703 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2"] Nov 02 14:10:58.876553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.876519 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cc8d39-ffc67f6d7-z47q2"] Nov 02 14:10:58.980274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:58.980238 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:59.150639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:59.150606 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:10:59.469095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:59.469014 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d30ac92-1b49-41e0-90fa-fe108d5679b8" path="/var/lib/kubelet/pods/6d30ac92-1b49-41e0-90fa-fe108d5679b8/volumes" Nov 02 14:10:59.469963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:10:59.469935 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3da4df8-c04e-42aa-bb51-df99e073b242" path="/var/lib/kubelet/pods/f3da4df8-c04e-42aa-bb51-df99e073b242/volumes" Nov 02 14:11:00.936017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:00.935127 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:01.074163 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:01.074126 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:11:01.462006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:01.461882 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:03.248774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:03.248604 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:03.529825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:03.529739 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:04.193599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.193434 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:04.385771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.385738 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd"] Nov 02 14:11:04.386368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386352 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f584b5b2-da4b-4a4e-be79-bdf70584d59f" containerName="payload-processing" Nov 02 14:11:04.386422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386370 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f584b5b2-da4b-4a4e-be79-bdf70584d59f" containerName="payload-processing" Nov 02 14:11:04.386422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386386 5591 cpu_manager.go:414] "containerMap: removing container" podUID="752361dc-c22c-4225-9f6e-17f1034004a2" containerName="payload-pre-processing" Nov 02 14:11:04.386422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386392 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752361dc-c22c-4225-9f6e-17f1034004a2" containerName="payload-pre-processing" Nov 02 14:11:04.386422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386404 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b8a9f158-2322-41bd-a21c-dd994ad10e74" containerName="maas-api" Nov 02 14:11:04.386422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386408 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a9f158-2322-41bd-a21c-dd994ad10e74" containerName="maas-api" Nov 02 14:11:04.386422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386420 5591 cpu_manager.go:414] "containerMap: removing container" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" Nov 02 14:11:04.386422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386425 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386444 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6d30ac92-1b49-41e0-90fa-fe108d5679b8" containerName="payload-pre-processing" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386449 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d30ac92-1b49-41e0-90fa-fe108d5679b8" containerName="payload-pre-processing" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386458 5591 cpu_manager.go:414] "containerMap: removing container" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386463 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386472 5591 cpu_manager.go:414] "containerMap: removing container" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386476 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386494 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386498 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386505 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f3da4df8-c04e-42aa-bb51-df99e073b242" containerName="payload-processing" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386510 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3da4df8-c04e-42aa-bb51-df99e073b242" containerName="payload-processing" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386518 5591 cpu_manager.go:414] "containerMap: removing container" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386522 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386593 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f584b5b2-da4b-4a4e-be79-bdf70584d59f" containerName="payload-processing" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386603 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="752361dc-c22c-4225-9f6e-17f1034004a2" containerName="payload-pre-processing" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386609 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a9f158-2322-41bd-a21c-dd994ad10e74" containerName="maas-api" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386614 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386623 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d30ac92-1b49-41e0-90fa-fe108d5679b8" containerName="payload-pre-processing" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386629 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c18551-b05a-4964-9964-dd5d7a79e943" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386636 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f128ca-a03d-471b-a951-4d19cddd1785" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386644 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8b9c04-2f85-4bed-96ce-7ea07e69317d" containerName="istio-proxy" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386649 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3da4df8-c04e-42aa-bb51-df99e073b242" containerName="payload-processing" Nov 02 14:11:04.386645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.386655 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ef2f4d-98a0-41d0-8a68-75a21356effb" containerName="istio-proxy" Nov 02 14:11:04.390194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.390175 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.410096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.410043 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-8acd34\"/\"kube-root-ca.crt\"" Nov 02 14:11:04.424393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.424368 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-8acd34\"/\"openshift-service-ca.crt\"" Nov 02 14:11:04.427745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.427720 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd"] Nov 02 14:11:04.430417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.430383 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-model-cache\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.430585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.430557 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tmp-dir\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.430660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.430616 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tls-certs\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.430725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.430657 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-dshm\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.430775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.430720 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxqz\" (UniqueName: \"kubernetes.io/projected/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-kube-api-access-scxqz\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.430855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.430778 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-home\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.440846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.440799 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:04.495630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.495552 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-8acd34\"/\"praxis-route-8acd34-kserve-self-signed-certs\"" Nov 02 14:11:04.496681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.496663 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-8acd34\"/\"default-dockercfg-2hp9r\"" Nov 02 14:11:04.531523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.531495 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tmp-dir\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.531671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.531537 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tls-certs\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.531671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.531642 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-dshm\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.531773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.531710 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxqz\" (UniqueName: \"kubernetes.io/projected/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-kube-api-access-scxqz\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.531773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.531764 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-home\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.531876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.531844 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-model-cache\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.531930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.531881 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tmp-dir\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.532158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.532136 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-home\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.532231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.532184 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-model-cache\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.533965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.533937 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-dshm\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.534088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.534069 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tls-certs\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.559737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.559709 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxqz\" (UniqueName: \"kubernetes.io/projected/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-kube-api-access-scxqz\") pod \"praxis-route-8acd34-kserve-545cbd4999-rljvd\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:04.659006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.656776 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:04.706007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:04.705248 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:05.058086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.058060 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd"] Nov 02 14:11:05.060347 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:11:05.060322 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9d9d6d4_dfde_46f4_a41a_c42b19255ed2.slice/crio-df8219c5b45d274f4895de8e6515b8f194b902baff6cede5b7d7ed676bbc6973 WatchSource:0}: Error finding container df8219c5b45d274f4895de8e6515b8f194b902baff6cede5b7d7ed676bbc6973: Status 404 returned error can't find the container with id df8219c5b45d274f4895de8e6515b8f194b902baff6cede5b7d7ed676bbc6973 Nov 02 14:11:05.135033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.127454 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:05.303008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.302953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:05.406764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.406730 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf"] Nov 02 14:11:05.407281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.407009 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" containerID="cri-o://2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1" gracePeriod=30 Nov 02 14:11:05.497542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.496953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:05.806366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.806263 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" event={"ID":"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2","Type":"ContainerStarted","Data":"6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50"} Nov 02 14:11:05.806366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.806320 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" event={"ID":"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2","Type":"ContainerStarted","Data":"df8219c5b45d274f4895de8e6515b8f194b902baff6cede5b7d7ed676bbc6973"} Nov 02 14:11:05.861037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:05.860965 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" podStartSLOduration=1.860950968 podStartE2EDuration="1.860950968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:05.85993148 +0000 UTC m=+1614.585553250" watchObservedRunningTime="2026-11-02 14:11:05.860950968 +0000 UTC m=+1614.586572738" Nov 02 14:11:06.389803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:06.389767 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw"] Nov 02 14:11:06.390043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:06.390018 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" containerID="cri-o://c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999" gracePeriod=30 Nov 02 14:11:06.705879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:06.705799 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:07.405853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:07.405696 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:08.679371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:08.679328 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:08.849689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:08.849657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:09.031258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.031168 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55"] Nov 02 14:11:09.032313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.032284 5591 cpu_manager.go:414] "containerMap: removing container" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" Nov 02 14:11:09.032459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.032447 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" Nov 02 14:11:09.032727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.032710 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="34932ab0-cb39-43e2-b9e1-ce8cb25b5e57" containerName="istio-proxy" Nov 02 14:11:09.036963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.036945 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.081766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.080668 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-de493d-dockercfg-7bsdd\"" Nov 02 14:11:09.081766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.080828 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-de493d\"" Nov 02 14:11:09.092034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.091973 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55"] Nov 02 14:11:09.153150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.153087 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8140329-c882-4896-af8f-42b082477459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55\" (UID: \"a8140329-c882-4896-af8f-42b082477459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.153471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.153441 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmjp\" (UniqueName: \"kubernetes.io/projected/a8140329-c882-4896-af8f-42b082477459-kube-api-access-dfmjp\") pod \"payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55\" (UID: \"a8140329-c882-4896-af8f-42b082477459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.170095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.170062 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm"] Nov 02 14:11:09.174173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.174149 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:09.197108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.197072 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm"] Nov 02 14:11:09.254565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.254529 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmjp\" (UniqueName: \"kubernetes.io/projected/a8140329-c882-4896-af8f-42b082477459-kube-api-access-dfmjp\") pod \"payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55\" (UID: \"a8140329-c882-4896-af8f-42b082477459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.254769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.254623 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm\" (UID: \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:09.254769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.254670 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmh6c\" (UniqueName: \"kubernetes.io/projected/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-kube-api-access-lmh6c\") pod \"payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm\" (UID: \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:09.254908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.254800 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8140329-c882-4896-af8f-42b082477459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55\" (UID: \"a8140329-c882-4896-af8f-42b082477459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.255476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.255451 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8140329-c882-4896-af8f-42b082477459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55\" (UID: \"a8140329-c882-4896-af8f-42b082477459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.281018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.280973 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmjp\" (UniqueName: \"kubernetes.io/projected/a8140329-c882-4896-af8f-42b082477459-kube-api-access-dfmjp\") pod \"payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55\" (UID: \"a8140329-c882-4896-af8f-42b082477459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.356050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.355930 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm\" (UID: \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:09.356050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.356034 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmh6c\" (UniqueName: \"kubernetes.io/projected/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-kube-api-access-lmh6c\") pod \"payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm\" (UID: \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:09.356672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.356651 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm\" (UID: \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:09.382178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.382144 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmh6c\" (UniqueName: \"kubernetes.io/projected/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-kube-api-access-lmh6c\") pod \"payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm\" (UID: \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:09.441907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.441878 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.490485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.490450 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:09.749609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.749582 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55"] Nov 02 14:11:09.752653 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:11:09.752627 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8140329_c882_4896_af8f_42b082477459.slice/crio-2447b9403d2b862013c7d362cba1418ea109e9b718bf51ff56f0e944689617b2 WatchSource:0}: Error finding container 2447b9403d2b862013c7d362cba1418ea109e9b718bf51ff56f0e944689617b2: Status 404 returned error can't find the container with id 2447b9403d2b862013c7d362cba1418ea109e9b718bf51ff56f0e944689617b2 Nov 02 14:11:09.837103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.837075 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm"] Nov 02 14:11:09.838562 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:11:09.838532 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bb89c0_6fa7_41cc_a59f_0bde5a4b9e8a.slice/crio-999e0079c54aa7dc3ea175ea3dd2efe6cf2c8ef5864fc47893d3e10f656d3cda WatchSource:0}: Error finding container 999e0079c54aa7dc3ea175ea3dd2efe6cf2c8ef5864fc47893d3e10f656d3cda: Status 404 returned error can't find the container with id 999e0079c54aa7dc3ea175ea3dd2efe6cf2c8ef5864fc47893d3e10f656d3cda Nov 02 14:11:09.846898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.846865 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" event={"ID":"a8140329-c882-4896-af8f-42b082477459","Type":"ContainerStarted","Data":"f3915b1b2f1e4ba25e11fb5b13c89fd89d15fd6c39b9ecc9abf246d1394428ce"} Nov 02 14:11:09.847050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.846930 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" event={"ID":"a8140329-c882-4896-af8f-42b082477459","Type":"ContainerStarted","Data":"2447b9403d2b862013c7d362cba1418ea109e9b718bf51ff56f0e944689617b2"} Nov 02 14:11:09.847050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.846959 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:09.923919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.923857 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:09.924317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:09.924258 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" podStartSLOduration=1.92424201 podStartE2EDuration="1.92424201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:09.922196806 +0000 UTC m=+1618.647818577" watchObservedRunningTime="2026-11-02 14:11:09.92424201 +0000 UTC m=+1618.649863781" Nov 02 14:11:10.677782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.677721 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:11:10.785964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.785928 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-token\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-token\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786000 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-envoy\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786088 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-certs\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786171 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-podinfo\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786231 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-credential-socket\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786369 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-data\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786413 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-socket\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786449 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-kube-api-access-q5kfd\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-kube-api-access-q5kfd\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786484 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82deb4cb-c2b7-4454-82d4-51a9f3959707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82deb4cb-c2b7-4454-82d4-51a9f3959707-istiod-ca-cert\") pod \"82deb4cb-c2b7-4454-82d4-51a9f3959707\" (UID: \"82deb4cb-c2b7-4454-82d4-51a9f3959707\") " Nov 02 14:11:10.786782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786663 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-certs" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:10.786782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786767 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-data" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:10.787019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.786809 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-socket" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:10.787157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.787135 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-credential-socket" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:10.787218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.787158 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82deb4cb-c2b7-4454-82d4-51a9f3959707-istiod-ca-cert" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:10.787218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.787206 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.787300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.787229 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.787300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.787253 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.788860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.788829 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-token" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:10.789256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.789223 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-podinfo" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:11:10.789509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.789476 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-kube-api-access-q5kfd" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "kube-api-access-q5kfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:10.790471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.790448 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-envoy" pod "82deb4cb-c2b7-4454-82d4-51a9f3959707" (UID: "82deb4cb-c2b7-4454-82d4-51a9f3959707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:10.856338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.856301 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" event={"ID":"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a","Type":"ContainerStarted","Data":"1421c99b4cdf1684ec3161296d112ceafcefd357f07123c69c7cc061aec7ec42"} Nov 02 14:11:10.856542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.856390 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" event={"ID":"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a","Type":"ContainerStarted","Data":"999e0079c54aa7dc3ea175ea3dd2efe6cf2c8ef5864fc47893d3e10f656d3cda"} Nov 02 14:11:10.856542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.856418 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:10.858227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.858200 5591 generic.go:356] "Generic (PLEG): container finished" podID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerID="2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1" exitCode=0 Nov 02 14:11:10.858348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.858242 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" Nov 02 14:11:10.858348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.858279 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" event={"ID":"82deb4cb-c2b7-4454-82d4-51a9f3959707","Type":"ContainerDied","Data":"2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1"} Nov 02 14:11:10.858348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.858326 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf" event={"ID":"82deb4cb-c2b7-4454-82d4-51a9f3959707","Type":"ContainerDied","Data":"3a38ec0f2520ee6fcf031fdf2813958ca67ed01579423df8769446ce2e1da006"} Nov 02 14:11:10.858348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.858349 5591 scope.go:122] "RemoveContainer" containerID="2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1" Nov 02 14:11:10.869331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.869311 5591 scope.go:122] "RemoveContainer" containerID="6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd" Nov 02 14:11:10.880616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.880593 5591 scope.go:122] "RemoveContainer" containerID="2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1" Nov 02 14:11:10.880938 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:10.880914 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1\": container with ID starting with 2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1 not found: ID does not exist" containerID="2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1" Nov 02 14:11:10.881068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.880945 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1"} err="failed to get container status \"2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1\": rpc error: code = NotFound desc = could not find container \"2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1\": container with ID starting with 2249d91af57c5aee8e3703aa82a9c9515f2d38d7e8f96fa6bf732fe5bc3a4ea1 not found: ID does not exist" Nov 02 14:11:10.881068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.880964 5591 scope.go:122] "RemoveContainer" containerID="6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd" Nov 02 14:11:10.881307 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:10.881287 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd\": container with ID starting with 6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd not found: ID does not exist" containerID="6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd" Nov 02 14:11:10.881352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.881315 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd"} err="failed to get container status \"6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd\": rpc error: code = NotFound desc = could not find container \"6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd\": container with ID starting with 6c0a508f698566ea68c3325dc852a35d5434839bb9795544e7c77d4d42c653dd not found: ID does not exist" Nov 02 14:11:10.888372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.888344 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5kfd\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-kube-api-access-q5kfd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.888473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.888374 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82deb4cb-c2b7-4454-82d4-51a9f3959707-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.888473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.888388 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.888473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.888401 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.888473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.888414 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82deb4cb-c2b7-4454-82d4-51a9f3959707-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.888473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.888427 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82deb4cb-c2b7-4454-82d4-51a9f3959707-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:10.910183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.910144 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" podStartSLOduration=1.9101339510000002 podStartE2EDuration="1.910133951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:10.90888199 +0000 UTC m=+1619.634503759" watchObservedRunningTime="2026-11-02 14:11:10.910133951 +0000 UTC m=+1619.635755723" Nov 02 14:11:10.952974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.952899 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf"] Nov 02 14:11:10.959014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:10.958969 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1d69c-openshift-default-7b5f9d8dd9-755zf"] Nov 02 14:11:11.434649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.434613 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" path="/var/lib/kubelet/pods/82deb4cb-c2b7-4454-82d4-51a9f3959707/volumes" Nov 02 14:11:11.680881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.680853 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:11:11.802568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.802536 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-podinfo\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.802664 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-socket\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.802703 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-credential-socket\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.802841 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-data\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.802867 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-token\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-token\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.802899 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-kube-api-access-rnmq2\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-kube-api-access-rnmq2\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.802951 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-envoy\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803024 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istiod-ca-cert\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803022 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-socket" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:11.803490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803084 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-certs\") pod \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\" (UID: \"43c0304a-3d10-4b71-93f2-fe27f62b0ebc\") " Nov 02 14:11:11.803490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803181 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-data" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:11.803638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803613 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-credential-socket" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:11.803705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803644 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istiod-ca-cert" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:11.803830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803811 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.803873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803837 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.803873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803845 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-certs" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:11.803873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.803852 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.805185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.805156 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-podinfo" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:11:11.805476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.805453 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-token" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:11.805476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.805465 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-kube-api-access-rnmq2" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "kube-api-access-rnmq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:11.805776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.805760 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-envoy" pod "43c0304a-3d10-4b71-93f2-fe27f62b0ebc" (UID: "43c0304a-3d10-4b71-93f2-fe27f62b0ebc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:11.865641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.865610 5591 generic.go:356] "Generic (PLEG): container finished" podID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerID="c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999" exitCode=0 Nov 02 14:11:11.865829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.865659 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" Nov 02 14:11:11.865829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.865695 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" event={"ID":"43c0304a-3d10-4b71-93f2-fe27f62b0ebc","Type":"ContainerDied","Data":"c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999"} Nov 02 14:11:11.865829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.865749 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw" event={"ID":"43c0304a-3d10-4b71-93f2-fe27f62b0ebc","Type":"ContainerDied","Data":"fca301fdd117eda58f436af962c01a40739fb792c00a4ccfe10886273219be3b"} Nov 02 14:11:11.865829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.865788 5591 scope.go:122] "RemoveContainer" containerID="c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999" Nov 02 14:11:11.876396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.876375 5591 scope.go:122] "RemoveContainer" containerID="c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999" Nov 02 14:11:11.876702 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:11.876682 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999\": container with ID starting with c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999 not found: ID does not exist" containerID="c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999" Nov 02 14:11:11.876800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.876711 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999"} err="failed to get container status \"c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999\": rpc error: code = NotFound desc = could not find container \"c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999\": container with ID starting with c4c9d8fb85fb585ce2b2ffbe82cff453fbecd7647ff29bdc27ddcc6fb41c6999 not found: ID does not exist" Nov 02 14:11:11.905297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.905266 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.905297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.905295 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmq2\" (UniqueName: \"kubernetes.io/projected/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-kube-api-access-rnmq2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.905505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.905312 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.905505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.905324 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.905505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.905335 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.905505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.905345 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c0304a-3d10-4b71-93f2-fe27f62b0ebc-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:11.958818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.958728 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw"] Nov 02 14:11:11.963629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:11.963600 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rqchw"] Nov 02 14:11:12.488941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:12.488719 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:13.431314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:13.431275 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" path="/var/lib/kubelet/pods/43c0304a-3d10-4b71-93f2-fe27f62b0ebc/volumes" Nov 02 14:11:14.706597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:14.706560 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:14.719571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:14.719542 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:14.796607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:14.791450 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:14.899183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:14.899155 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:11:16.583010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:16.580714 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:18.426073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:18.426034 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:20.303304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:20.303266 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:20.860435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:20.860404 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:11:21.081209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:21.076867 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:21.869390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:21.869359 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:11:22.910098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:22.908581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:23.076409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:23.076374 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:24.893938 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:24.893902 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:25.586588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:25.586522 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:25.770388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:25.770351 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:26.572804 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:26.572743 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:26.910403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:26.910362 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8"] Nov 02 14:11:26.910678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:26.910627 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" containerID="cri-o://3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5" gracePeriod=30 Nov 02 14:11:27.036965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:27.036912 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:27.259405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:27.259321 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:27.920041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:27.920003 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:29.003851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:29.003822 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-t792x_566435d0-5ce9-4ef4-9a61-9ebb69b8d17d/payload-processing/0.log" Nov 02 14:11:29.232659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:29.232624 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-t792x_566435d0-5ce9-4ef4-9a61-9ebb69b8d17d/payload-processing/0.log" Nov 02 14:11:29.422825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:29.422795 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm_12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a/payload-processing/0.log" Nov 02 14:11:29.910805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:29.910762 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:30.430485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:30.430438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:31.074640 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:31.074598 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:11:31.875726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:31.875618 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:32.192334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.192306 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:11:32.333656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.333626 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-data\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.333833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.333671 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9b8850-3749-47d0-b47e-f77f649f77d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b8850-3749-47d0-b47e-f77f649f77d3-istiod-ca-cert\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.333833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.333733 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-token\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.333833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.333784 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-certs\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.334033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.333913 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-credential-socket\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.334033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.333925 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-data" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:32.334160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334034 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-podinfo\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.334160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334068 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-certs" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:32.334160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334109 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-envoy\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.334312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334191 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-credential-socket" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:32.334312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334210 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9b8850-3749-47d0-b47e-f77f649f77d3-istiod-ca-cert" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:32.334312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334241 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-socket\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.334312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334290 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-kube-api-access-88qbh\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-kube-api-access-88qbh\") pod \"ce9b8850-3749-47d0-b47e-f77f649f77d3\" (UID: \"ce9b8850-3749-47d0-b47e-f77f649f77d3\") " Nov 02 14:11:32.334507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334491 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-socket" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:32.334880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334854 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:32.334880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334882 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:32.334880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334897 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9b8850-3749-47d0-b47e-f77f649f77d3-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:32.335168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334911 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:32.335168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.334924 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:32.336120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.336095 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-token" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:32.336212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.336172 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-envoy" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:32.336257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.336225 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-podinfo" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:11:32.336487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.336467 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-kube-api-access-88qbh" pod "ce9b8850-3749-47d0-b47e-f77f649f77d3" (UID: "ce9b8850-3749-47d0-b47e-f77f649f77d3"). InnerVolumeSpecName "kube-api-access-88qbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:32.436722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.436644 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qbh\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-kube-api-access-88qbh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:32.436722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.436677 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:32.436722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.436687 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:32.436722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:32.436696 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9b8850-3749-47d0-b47e-f77f649f77d3-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:33.017367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.017324 5591 generic.go:356] "Generic (PLEG): container finished" podID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerID="3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5" exitCode=0 Nov 02 14:11:33.017785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.017379 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" event={"ID":"ce9b8850-3749-47d0-b47e-f77f649f77d3","Type":"ContainerDied","Data":"3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5"} Nov 02 14:11:33.017785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.017417 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" event={"ID":"ce9b8850-3749-47d0-b47e-f77f649f77d3","Type":"ContainerDied","Data":"0c4b744430cc66fda86dbf71049786a6e93ce75536a78c28b307b674836487a8"} Nov 02 14:11:33.017785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.017437 5591 scope.go:122] "RemoveContainer" containerID="3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5" Nov 02 14:11:33.017785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.017379 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8" Nov 02 14:11:33.028126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.028027 5591 scope.go:122] "RemoveContainer" containerID="7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a" Nov 02 14:11:33.036872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.036734 5591 scope.go:122] "RemoveContainer" containerID="3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5" Nov 02 14:11:33.037014 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:33.036995 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5\": container with ID starting with 3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5 not found: ID does not exist" containerID="3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5" Nov 02 14:11:33.037061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.037022 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5"} err="failed to get container status \"3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5\": rpc error: code = NotFound desc = could not find container \"3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5\": container with ID starting with 3c2dc12eeef7e9b53a000c427c5f8d649fc5b03082b283210dba54d70c2c13b5 not found: ID does not exist" Nov 02 14:11:33.037061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.037042 5591 scope.go:122] "RemoveContainer" containerID="7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a" Nov 02 14:11:33.037268 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:33.037253 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a\": container with ID starting with 7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a not found: ID does not exist" containerID="7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a" Nov 02 14:11:33.037306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.037272 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a"} err="failed to get container status \"7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a\": rpc error: code = NotFound desc = could not find container \"7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a\": container with ID starting with 7eeb57def7ef7671c872f9fe348701913237a63280ab5186dffe852cf3a54d6a not found: ID does not exist" Nov 02 14:11:33.092540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.092505 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8"] Nov 02 14:11:33.106962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.106926 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-cc8d39-openshift-default-8488cc4549-5t8p8"] Nov 02 14:11:33.365443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.365358 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m"] Nov 02 14:11:33.434651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.434616 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" path="/var/lib/kubelet/pods/ce9b8850-3749-47d0-b47e-f77f649f77d3/volumes" Nov 02 14:11:33.549181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.549149 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:33.742543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.742501 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:33.905166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:33.905134 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:34.092084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:34.091997 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:35.516403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:35.516375 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:36.993438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:36.993396 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:37.761862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:37.761815 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:37.882036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:37.881975 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj"] Nov 02 14:11:37.882296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:37.882261 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" podUID="f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b" containerName="payload-pre-processing" containerID="cri-o://6ed8b4f08ff732a8f6c45abf3d8b2c9f27744b962ddc72441232ddcd7fb42be4" gracePeriod=30 Nov 02 14:11:38.018380 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.018302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:38.077529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.077492 5591 generic.go:356] "Generic (PLEG): container finished" podID="f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b" containerID="6ed8b4f08ff732a8f6c45abf3d8b2c9f27744b962ddc72441232ddcd7fb42be4" exitCode=0 Nov 02 14:11:38.077857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.077833 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" event={"ID":"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b","Type":"ContainerDied","Data":"6ed8b4f08ff732a8f6c45abf3d8b2c9f27744b962ddc72441232ddcd7fb42be4"} Nov 02 14:11:38.158616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.158591 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:11:38.236006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.234965 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:38.365056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.360525 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-kube-api-access-lq6tn\" (UniqueName: \"kubernetes.io/projected/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-kube-api-access-lq6tn\") pod \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\" (UID: \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\") " Nov 02 14:11:38.365056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.360695 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-plugins-config-volume\") pod \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\" (UID: \"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b\") " Nov 02 14:11:38.365056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.363840 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-plugins-config-volume" pod "f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b" (UID: "f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:38.368032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.366282 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" podUID="8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6" containerName="payload-processing" containerID="cri-o://17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7" gracePeriod=25 Nov 02 14:11:38.368032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.367175 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-kube-api-access-lq6tn" pod "f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b" (UID: "f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b"). InnerVolumeSpecName "kube-api-access-lq6tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:38.462867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.462828 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:38.462867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.462863 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq6tn\" (UniqueName: \"kubernetes.io/projected/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b-kube-api-access-lq6tn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:38.644886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.644855 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:11:38.664936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.664905 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-kube-api-access-dvvtx\" (UniqueName: \"kubernetes.io/projected/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-kube-api-access-dvvtx\") pod \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\" (UID: \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\") " Nov 02 14:11:38.665093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.665060 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-plugins-config-volume\") pod \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\" (UID: \"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6\") " Nov 02 14:11:38.665549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.665517 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-plugins-config-volume" pod "8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6" (UID: "8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:38.665696 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.665633 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:38.667694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.667657 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-kube-api-access-dvvtx" pod "8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6" (UID: "8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6"). InnerVolumeSpecName "kube-api-access-dvvtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:38.727231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.727185 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:38.767297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:38.767262 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvvtx\" (UniqueName: \"kubernetes.io/projected/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6-kube-api-access-dvvtx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:39.085386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.085355 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" Nov 02 14:11:39.085811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.085352 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj" event={"ID":"f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b","Type":"ContainerDied","Data":"50d3acbdbaff3abe1c82d2e1368eca11f465245da503ae4074f02d8f8422154d"} Nov 02 14:11:39.085811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.085494 5591 scope.go:122] "RemoveContainer" containerID="6ed8b4f08ff732a8f6c45abf3d8b2c9f27744b962ddc72441232ddcd7fb42be4" Nov 02 14:11:39.086896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.086866 5591 generic.go:356] "Generic (PLEG): container finished" podID="8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6" containerID="17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7" exitCode=0 Nov 02 14:11:39.086971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.086908 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" Nov 02 14:11:39.086971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.086908 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" event={"ID":"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6","Type":"ContainerDied","Data":"17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7"} Nov 02 14:11:39.086971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.086959 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m" event={"ID":"8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6","Type":"ContainerDied","Data":"af0f12da96855cca63b02c0f9cf4e64ac8be1070f2b19424c84ff199857bc004"} Nov 02 14:11:39.096007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.095960 5591 scope.go:122] "RemoveContainer" containerID="17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7" Nov 02 14:11:39.105145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.105129 5591 scope.go:122] "RemoveContainer" containerID="17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7" Nov 02 14:11:39.105410 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:39.105390 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7\": container with ID starting with 17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7 not found: ID does not exist" containerID="17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7" Nov 02 14:11:39.105468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.105422 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7"} err="failed to get container status \"17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7\": rpc error: code = NotFound desc = could not find container \"17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7\": container with ID starting with 17a1065c4f1ee6027a2ae6d3d1f4d66f42d285e1c3448559aab24f023288f7f7 not found: ID does not exist" Nov 02 14:11:39.146943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.146908 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m"] Nov 02 14:11:39.161536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.161465 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-370905-74c5c5fd77-cpb5m"] Nov 02 14:11:39.196172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.196146 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj"] Nov 02 14:11:39.198881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.198861 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-13367a-5899cc779-qngdj"] Nov 02 14:11:39.432542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.432467 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6" path="/var/lib/kubelet/pods/8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6/volumes" Nov 02 14:11:39.433042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:39.433024 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b" path="/var/lib/kubelet/pods/f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b/volumes" Nov 02 14:11:40.947007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:40.940087 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:41.278774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:41.278678 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:11:41.368010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:41.361582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:42.712137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:42.712100 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:43.629453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:43.629408 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:43.837216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:43.837183 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:44.938009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:44.932223 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:45.108214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:45.108178 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:45.501528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:45.501483 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6"] Nov 02 14:11:45.501730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:45.501693 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" containerID="cri-o://8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54" gracePeriod=30 Nov 02 14:11:46.002778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:46.002741 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:46.194259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:46.194226 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:46.315137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:46.315052 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98"] Nov 02 14:11:46.315340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:46.315298 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" containerID="cri-o://a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738" gracePeriod=30 Nov 02 14:11:46.402677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:46.402625 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:47.172918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:47.172887 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:47.883623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:47.883578 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:49.883031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:49.882969 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:50.777843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.777778 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 14:11:50.817799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.817762 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5454e3c0-401e-4a75-9602-729f031c98d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5454e3c0-401e-4a75-9602-729f031c98d6-istio-podinfo\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.817848 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-socket\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.817945 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-istio-token\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-istio-token\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818030 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-envoy\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818053 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-credential-socket\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818081 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-socket" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:50.818289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818137 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-kube-api-access-zbqns\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-kube-api-access-zbqns\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818208 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-certs\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818244 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-data\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818280 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5454e3c0-401e-4a75-9602-729f031c98d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5454e3c0-401e-4a75-9602-729f031c98d6-istiod-ca-cert\") pod \"5454e3c0-401e-4a75-9602-729f031c98d6\" (UID: \"5454e3c0-401e-4a75-9602-729f031c98d6\") " Nov 02 14:11:50.818498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818355 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-credential-socket" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:50.818670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818557 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-data" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:50.818670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818633 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-certs" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:50.818900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.818864 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5454e3c0-401e-4a75-9602-729f031c98d6-istiod-ca-cert" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:50.819059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.819034 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.819149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.819065 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.819149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.819079 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5454e3c0-401e-4a75-9602-729f031c98d6-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.819149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.819092 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.819149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.819106 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.820298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.820266 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-istio-token" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:50.820458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.820438 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5454e3c0-401e-4a75-9602-729f031c98d6-istio-podinfo" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:11:50.820504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.820451 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-envoy" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:50.820504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.820466 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-kube-api-access-zbqns" pod "5454e3c0-401e-4a75-9602-729f031c98d6" (UID: "5454e3c0-401e-4a75-9602-729f031c98d6"). InnerVolumeSpecName "kube-api-access-zbqns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:50.919974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.919943 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.919974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.919974 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5454e3c0-401e-4a75-9602-729f031c98d6-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.919974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.920001 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbqns\" (UniqueName: \"kubernetes.io/projected/5454e3c0-401e-4a75-9602-729f031c98d6-kube-api-access-zbqns\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.920440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.920015 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5454e3c0-401e-4a75-9602-729f031c98d6-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.998778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:50.998745 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:51.167085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.167051 5591 generic.go:356] "Generic (PLEG): container finished" podID="5454e3c0-401e-4a75-9602-729f031c98d6" containerID="8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54" exitCode=0 Nov 02 14:11:51.167264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.167100 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" Nov 02 14:11:51.167264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.167129 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" event={"ID":"5454e3c0-401e-4a75-9602-729f031c98d6","Type":"ContainerDied","Data":"8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54"} Nov 02 14:11:51.167264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.167184 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6" event={"ID":"5454e3c0-401e-4a75-9602-729f031c98d6","Type":"ContainerDied","Data":"1a8de16bd7fe0ae7cefc20d91d25c723201013008a2ee38a06dc4c83119bcd92"} Nov 02 14:11:51.167264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.167206 5591 scope.go:122] "RemoveContainer" containerID="8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54" Nov 02 14:11:51.186216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.186196 5591 scope.go:122] "RemoveContainer" containerID="8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54" Nov 02 14:11:51.186483 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:51.186461 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54\": container with ID starting with 8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54 not found: ID does not exist" containerID="8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54" Nov 02 14:11:51.186537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.186491 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54"} err="failed to get container status \"8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54\": rpc error: code = NotFound desc = could not find container \"8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54\": container with ID starting with 8cc78c306ea82465773f7cc315e0047f12afa12dd3978745c19b1b6b16576b54 not found: ID does not exist" Nov 02 14:11:51.267109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.267077 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6"] Nov 02 14:11:51.282030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.281997 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-4m6w6"] Nov 02 14:11:51.431536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.431452 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" path="/var/lib/kubelet/pods/5454e3c0-401e-4a75-9602-729f031c98d6/volumes" Nov 02 14:11:51.597605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.597579 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:11:51.633617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.633587 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-credential-socket\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.633757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.633657 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-certs\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.633757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.633709 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-data\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.633836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.633769 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-token\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-token\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.633836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.633807 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istiod-ca-cert\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.633900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.633861 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-kube-api-access-vlb2n\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-kube-api-access-vlb2n\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.633945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.633892 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-credential-socket" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:51.633945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.633928 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-envoy\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.634075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634025 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-socket\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.634075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634029 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-certs" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:51.634075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634050 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-podinfo\") pod \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\" (UID: \"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc\") " Nov 02 14:11:51.634240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634115 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-data" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:51.634240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634228 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istiod-ca-cert" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:51.634378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634357 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-socket" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:51.634721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634693 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.634835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634723 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.634835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634740 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.634835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634755 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.634835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.634768 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.636167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.636140 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-envoy" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:11:51.636257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.636227 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-token" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:51.636257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.636229 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-kube-api-access-vlb2n" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "kube-api-access-vlb2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:51.636335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.636287 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-podinfo" pod "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" (UID: "c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:11:51.735768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.735690 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.735768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.735719 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.735768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.735728 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlb2n\" (UniqueName: \"kubernetes.io/projected/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-kube-api-access-vlb2n\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.735768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.735737 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:11:51.880756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:51.880621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:52.175116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.175077 5591 generic.go:356] "Generic (PLEG): container finished" podID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerID="a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738" exitCode=0 Nov 02 14:11:52.175610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.175138 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" Nov 02 14:11:52.175610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.175154 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" event={"ID":"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc","Type":"ContainerDied","Data":"a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738"} Nov 02 14:11:52.175610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.175208 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98" event={"ID":"c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc","Type":"ContainerDied","Data":"d91825da94fd21ae328542a55ec1a57552d1d96fc6a22a331fafee82d0d67207"} Nov 02 14:11:52.175610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.175237 5591 scope.go:122] "RemoveContainer" containerID="a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738" Nov 02 14:11:52.186926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.186905 5591 scope.go:122] "RemoveContainer" containerID="362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199" Nov 02 14:11:52.195486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.195455 5591 scope.go:122] "RemoveContainer" containerID="a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738" Nov 02 14:11:52.195753 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:52.195733 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738\": container with ID starting with a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738 not found: ID does not exist" containerID="a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738" Nov 02 14:11:52.195820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.195759 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738"} err="failed to get container status \"a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738\": rpc error: code = NotFound desc = could not find container \"a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738\": container with ID starting with a99662a385549e56f4ae9789e0979295edb5ea2bc544b7c7fb864003ee80f738 not found: ID does not exist" Nov 02 14:11:52.195820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.195777 5591 scope.go:122] "RemoveContainer" containerID="362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199" Nov 02 14:11:52.196044 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:11:52.196020 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199\": container with ID starting with 362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199 not found: ID does not exist" containerID="362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199" Nov 02 14:11:52.196084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.196052 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199"} err="failed to get container status \"362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199\": rpc error: code = NotFound desc = could not find container \"362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199\": container with ID starting with 362d9a55da8a6391a05fc69a13d881fa5b5a34133a627ddb6ee6e46b91a38199 not found: ID does not exist" Nov 02 14:11:52.237864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.237821 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98"] Nov 02 14:11:52.252345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:52.252298 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-13367a-openshift-default-5df578dbfc-gnz98"] Nov 02 14:11:53.432159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:53.432127 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" path="/var/lib/kubelet/pods/c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc/volumes" Nov 02 14:11:53.814880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:53.814787 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:53.985058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:53.985024 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:56.259008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:56.241007 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:57.374621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:57.374581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:57.885969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:57.885925 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:11:58.515397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:11:58.515366 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:00.020133 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.020092 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:00.054300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.054262 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm"] Nov 02 14:12:00.054565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.054539 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" podUID="12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a" containerName="payload-processing" containerID="cri-o://1421c99b4cdf1684ec3161296d112ceafcefd357f07123c69c7cc061aec7ec42" gracePeriod=30 Nov 02 14:12:00.091458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.091422 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55"] Nov 02 14:12:00.091724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.091691 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" podUID="a8140329-c882-4896-af8f-42b082477459" containerName="payload-pre-processing" containerID="cri-o://f3915b1b2f1e4ba25e11fb5b13c89fd89d15fd6c39b9ecc9abf246d1394428ce" gracePeriod=30 Nov 02 14:12:00.236450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.236409 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:00.237810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.237780 5591 generic.go:356] "Generic (PLEG): container finished" podID="a8140329-c882-4896-af8f-42b082477459" containerID="f3915b1b2f1e4ba25e11fb5b13c89fd89d15fd6c39b9ecc9abf246d1394428ce" exitCode=0 Nov 02 14:12:00.237948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.237872 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" event={"ID":"a8140329-c882-4896-af8f-42b082477459","Type":"ContainerDied","Data":"f3915b1b2f1e4ba25e11fb5b13c89fd89d15fd6c39b9ecc9abf246d1394428ce"} Nov 02 14:12:00.239503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.239471 5591 generic.go:356] "Generic (PLEG): container finished" podID="12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a" containerID="1421c99b4cdf1684ec3161296d112ceafcefd357f07123c69c7cc061aec7ec42" exitCode=0 Nov 02 14:12:00.239624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.239611 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" event={"ID":"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a","Type":"ContainerDied","Data":"1421c99b4cdf1684ec3161296d112ceafcefd357f07123c69c7cc061aec7ec42"} Nov 02 14:12:00.355363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.355323 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9"] Nov 02 14:12:00.355875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.355817 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" podUID="6e58081c-1e1b-4aee-b25d-f012110fc2dd" containerName="maas-api" containerID="cri-o://94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57" gracePeriod=30 Nov 02 14:12:00.412896 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:00.412854 5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e58081c_1e1b_4aee_b25d_f012110fc2dd.slice/crio-conmon-94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:12:00.414002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.413954 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:00.438367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.438338 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:12:00.469657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.469633 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:12:00.549838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.549803 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8140329-c882-4896-af8f-42b082477459-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8140329-c882-4896-af8f-42b082477459-plugins-config-volume\") pod \"a8140329-c882-4896-af8f-42b082477459\" (UID: \"a8140329-c882-4896-af8f-42b082477459\") " Nov 02 14:12:00.550574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.550045 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-kube-api-access-lmh6c\" (UniqueName: \"kubernetes.io/projected/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-kube-api-access-lmh6c\") pod \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\" (UID: \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\") " Nov 02 14:12:00.550574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.550115 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8140329-c882-4896-af8f-42b082477459-kube-api-access-dfmjp\" (UniqueName: \"kubernetes.io/projected/a8140329-c882-4896-af8f-42b082477459-kube-api-access-dfmjp\") pod \"a8140329-c882-4896-af8f-42b082477459\" (UID: \"a8140329-c882-4896-af8f-42b082477459\") " Nov 02 14:12:00.550574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.550161 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-plugins-config-volume\") pod \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\" (UID: \"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a\") " Nov 02 14:12:00.550574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.550237 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8140329-c882-4896-af8f-42b082477459-plugins-config-volume" pod "a8140329-c882-4896-af8f-42b082477459" (UID: "a8140329-c882-4896-af8f-42b082477459"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:00.550574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.550548 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8140329-c882-4896-af8f-42b082477459-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.551394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.551228 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-plugins-config-volume" pod "12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a" (UID: "12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:00.559056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.557238 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-kube-api-access-lmh6c" pod "12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a" (UID: "12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a"). InnerVolumeSpecName "kube-api-access-lmh6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:00.559684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.559657 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8140329-c882-4896-af8f-42b082477459-kube-api-access-dfmjp" pod "a8140329-c882-4896-af8f-42b082477459" (UID: "a8140329-c882-4896-af8f-42b082477459"). InnerVolumeSpecName "kube-api-access-dfmjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:00.614554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.613671 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:00.645543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.645516 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:12:00.652200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.652172 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmh6c\" (UniqueName: \"kubernetes.io/projected/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-kube-api-access-lmh6c\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.652200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.652200 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfmjp\" (UniqueName: \"kubernetes.io/projected/a8140329-c882-4896-af8f-42b082477459-kube-api-access-dfmjp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.652361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.652215 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.752614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.752581 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-metrics-certs\") pod \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " Nov 02 14:12:00.752816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.752651 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e58081c-1e1b-4aee-b25d-f012110fc2dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e58081c-1e1b-4aee-b25d-f012110fc2dd-openshift-service-ca\") pod \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " Nov 02 14:12:00.752816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.752720 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e58081c-1e1b-4aee-b25d-f012110fc2dd-kube-api-access-wjwqm\" (UniqueName: \"kubernetes.io/projected/6e58081c-1e1b-4aee-b25d-f012110fc2dd-kube-api-access-wjwqm\") pod \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " Nov 02 14:12:00.752943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.752847 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls\") pod \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\" (UID: \"6e58081c-1e1b-4aee-b25d-f012110fc2dd\") " Nov 02 14:12:00.753069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.753036 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e58081c-1e1b-4aee-b25d-f012110fc2dd-openshift-service-ca" pod "6e58081c-1e1b-4aee-b25d-f012110fc2dd" (UID: "6e58081c-1e1b-4aee-b25d-f012110fc2dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:00.753350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.753329 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e58081c-1e1b-4aee-b25d-f012110fc2dd-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.754855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.754824 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls" pod "6e58081c-1e1b-4aee-b25d-f012110fc2dd" (UID: "6e58081c-1e1b-4aee-b25d-f012110fc2dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:00.754946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.754872 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e58081c-1e1b-4aee-b25d-f012110fc2dd-kube-api-access-wjwqm" pod "6e58081c-1e1b-4aee-b25d-f012110fc2dd" (UID: "6e58081c-1e1b-4aee-b25d-f012110fc2dd"). InnerVolumeSpecName "kube-api-access-wjwqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:00.754946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.754872 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-metrics-certs" pod "6e58081c-1e1b-4aee-b25d-f012110fc2dd" (UID: "6e58081c-1e1b-4aee-b25d-f012110fc2dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:00.781810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.781778 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:00.871970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.871514 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.871970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.871548 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e58081c-1e1b-4aee-b25d-f012110fc2dd-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.871970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:00.871563 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjwqm\" (UniqueName: \"kubernetes.io/projected/6e58081c-1e1b-4aee-b25d-f012110fc2dd-kube-api-access-wjwqm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:01.079219 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:01.079149 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:12:01.247453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.247428 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" Nov 02 14:12:01.247453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.247435 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55" event={"ID":"a8140329-c882-4896-af8f-42b082477459","Type":"ContainerDied","Data":"2447b9403d2b862013c7d362cba1418ea109e9b718bf51ff56f0e944689617b2"} Nov 02 14:12:01.247725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.247515 5591 scope.go:122] "RemoveContainer" containerID="f3915b1b2f1e4ba25e11fb5b13c89fd89d15fd6c39b9ecc9abf246d1394428ce" Nov 02 14:12:01.249114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.249096 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" Nov 02 14:12:01.249206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.249175 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm" event={"ID":"12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a","Type":"ContainerDied","Data":"999e0079c54aa7dc3ea175ea3dd2efe6cf2c8ef5864fc47893d3e10f656d3cda"} Nov 02 14:12:01.250789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.250766 5591 generic.go:356] "Generic (PLEG): container finished" podID="6e58081c-1e1b-4aee-b25d-f012110fc2dd" containerID="94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57" exitCode=0 Nov 02 14:12:01.250918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.250821 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" Nov 02 14:12:01.250918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.250854 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" event={"ID":"6e58081c-1e1b-4aee-b25d-f012110fc2dd","Type":"ContainerDied","Data":"94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57"} Nov 02 14:12:01.250918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.250892 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9" event={"ID":"6e58081c-1e1b-4aee-b25d-f012110fc2dd","Type":"ContainerDied","Data":"d936212ce23217eb2db28a8e4837641ec0403aee424bcd0ab3f665c5fa041877"} Nov 02 14:12:01.262093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.262067 5591 scope.go:122] "RemoveContainer" containerID="1421c99b4cdf1684ec3161296d112ceafcefd357f07123c69c7cc061aec7ec42" Nov 02 14:12:01.272304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.272286 5591 scope.go:122] "RemoveContainer" containerID="94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57" Nov 02 14:12:01.281215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.281197 5591 scope.go:122] "RemoveContainer" containerID="94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57" Nov 02 14:12:01.281487 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:01.281467 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57\": container with ID starting with 94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57 not found: ID does not exist" containerID="94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57" Nov 02 14:12:01.281542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.281495 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57"} err="failed to get container status \"94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57\": rpc error: code = NotFound desc = could not find container \"94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57\": container with ID starting with 94989e5653227cc3e27377577265bfe62851b245145c2667b4cad00081342b57 not found: ID does not exist" Nov 02 14:12:01.341511 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.341461 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:01.354745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.354713 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55"] Nov 02 14:12:01.357865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.357838 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de493d-6bc55bfd55-hts55"] Nov 02 14:12:01.397480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.397447 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9"] Nov 02 14:12:01.402074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.402033 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-794667-5db5df4f-c8vq9"] Nov 02 14:12:01.431876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.431844 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e58081c-1e1b-4aee-b25d-f012110fc2dd" path="/var/lib/kubelet/pods/6e58081c-1e1b-4aee-b25d-f012110fc2dd/volumes" Nov 02 14:12:01.432358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.432341 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8140329-c882-4896-af8f-42b082477459" path="/var/lib/kubelet/pods/a8140329-c882-4896-af8f-42b082477459/volumes" Nov 02 14:12:01.441266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.441232 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm"] Nov 02 14:12:01.443443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:01.443416 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de493d-6ffd477d66-wb5sm"] Nov 02 14:12:03.009610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:03.009382 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:03.431251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:03.431211 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a" path="/var/lib/kubelet/pods/12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a/volumes" Nov 02 14:12:04.015785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:04.015732 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:04.183788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:04.183751 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:04.954569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:04.954531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:05.131046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:05.131010 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:05.327510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:05.326871 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:06.824510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:06.824455 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:07.823362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:07.823325 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:08.006487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:08.006450 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:08.608152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:08.608119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:08.776814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:08.776780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:09.131013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:09.128803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:09.666010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:09.665921 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:09.844730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:09.844692 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:10.517275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:10.517239 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:10.723077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:10.723042 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:10.799662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:10.799585 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2"] Nov 02 14:12:10.799835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:10.799809 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" containerID="cri-o://b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740" gracePeriod=30 Nov 02 14:12:10.921171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:10.921122 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:11.123495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:11.123406 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:12.363246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.363212 5591 scope.go:122] "RemoveContainer" containerID="a774cc9237a4843827c2e00371cad31b8457bad82ad267d19155a91495fe9387" Nov 02 14:12:12.373188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.373157 5591 scope.go:122] "RemoveContainer" containerID="582539ea4babe74343ee51a8e657437a351133af405ab1a9ab0bd956cbc9f7cf" Nov 02 14:12:12.449463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.449436 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:12.502304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.502253 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v"] Nov 02 14:12:12.509097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.503375 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" Nov 02 14:12:12.509368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509342 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" Nov 02 14:12:12.509509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509493 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b" containerName="payload-pre-processing" Nov 02 14:12:12.509628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509601 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b" containerName="payload-pre-processing" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509640 5591 cpu_manager.go:414] "containerMap: removing container" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509651 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509679 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509687 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509705 5591 cpu_manager.go:414] "containerMap: removing container" podUID="12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a" containerName="payload-processing" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509713 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a" containerName="payload-processing" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509746 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6" containerName="payload-processing" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509753 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6" containerName="payload-processing" Nov 02 14:12:12.509767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509766 5591 cpu_manager.go:414] "containerMap: removing container" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509773 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509785 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6e58081c-1e1b-4aee-b25d-f012110fc2dd" containerName="maas-api" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509792 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e58081c-1e1b-4aee-b25d-f012110fc2dd" containerName="maas-api" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509812 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509819 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509835 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509842 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509855 5591 cpu_manager.go:414] "containerMap: removing container" podUID="a8140329-c882-4896-af8f-42b082477459" containerName="payload-pre-processing" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.509864 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8140329-c882-4896-af8f-42b082477459" containerName="payload-pre-processing" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510057 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510071 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510084 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bfd415-5fa9-4cf8-a26f-35a1262f7b1b" containerName="payload-pre-processing" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510093 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510105 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510116 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bb89c0-6fa7-41cc-a59f-0bde5a4b9e8a" containerName="payload-processing" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510130 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f327b74-2f0c-4d9c-9ac6-c5eaf6cbe0f6" containerName="payload-processing" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510141 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c0304a-3d10-4b71-93f2-fe27f62b0ebc" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510150 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e58081c-1e1b-4aee-b25d-f012110fc2dd" containerName="maas-api" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510159 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5454e3c0-401e-4a75-9602-729f031c98d6" containerName="istio-proxy" Nov 02 14:12:12.510241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.510171 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8140329-c882-4896-af8f-42b082477459" containerName="payload-pre-processing" Nov 02 14:12:12.516267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.516245 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.532438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.532413 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v"] Nov 02 14:12:12.640899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.640871 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:12.675358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675324 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b605f12-2143-4e18-8a4d-87945d237404-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.675531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675367 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.675531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675494 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.675618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675538 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.675668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675652 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b605f12-2143-4e18-8a4d-87945d237404-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.675712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675681 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzs8d\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-kube-api-access-tzs8d\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.675764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675715 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.675868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675843 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.675934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.675905 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.776860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.776823 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b605f12-2143-4e18-8a4d-87945d237404-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.776879 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzs8d\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-kube-api-access-tzs8d\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.776915 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.776943 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777130 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777247 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b605f12-2143-4e18-8a4d-87945d237404-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777286 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777360 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777393 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777413 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777512 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777652 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.777938 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777917 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b605f12-2143-4e18-8a4d-87945d237404-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.778058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.777917 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.779844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.779821 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b605f12-2143-4e18-8a4d-87945d237404-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.779951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.779909 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.804091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.804068 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzs8d\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-kube-api-access-tzs8d\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.804213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.804153 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:12.831750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:12.831719 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:13.010617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.009839 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:13.207147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.207056 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:13.209700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.209674 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v"] Nov 02 14:12:13.210540 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:12:13.210517 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b605f12_2143_4e18_8a4d_87945d237404.slice/crio-5368f48358b966dd5d9c30c955b62fa9436accddd7c00106d9ee7a4ffbfda244 WatchSource:0}: Error finding container 5368f48358b966dd5d9c30c955b62fa9436accddd7c00106d9ee7a4ffbfda244: Status 404 returned error can't find the container with id 5368f48358b966dd5d9c30c955b62fa9436accddd7c00106d9ee7a4ffbfda244 Nov 02 14:12:13.212956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.212928 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:13.213029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.213012 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:13.213066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.213046 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:13.331062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.331020 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" event={"ID":"4b605f12-2143-4e18-8a4d-87945d237404","Type":"ContainerStarted","Data":"2fd38e9a358b501f9b2356e49db93c0601fc7aeba0681bd5beca34e3bb0352af"} Nov 02 14:12:13.331236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.331086 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" event={"ID":"4b605f12-2143-4e18-8a4d-87945d237404","Type":"ContainerStarted","Data":"5368f48358b966dd5d9c30c955b62fa9436accddd7c00106d9ee7a4ffbfda244"} Nov 02 14:12:13.386687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.386630 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" podStartSLOduration=1.386616462 podStartE2EDuration="1.386616462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:13.383028924 +0000 UTC m=+1682.108650694" watchObservedRunningTime="2026-11-02 14:12:13.386616462 +0000 UTC m=+1682.112238231" Nov 02 14:12:13.831838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:13.831801 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:14.849007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:14.848644 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:12:14.849007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:14.848699 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:12:15.063118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:15.063063 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:15.322444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:15.322414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:15.490210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:15.490176 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:15.837007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:15.833344 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:12:15.837007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:15.833404 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:12:16.112934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.112904 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:12:16.129639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129607 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-socket\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.129770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129660 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92939b14-a553-47be-9bd8-a5de527a88ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92939b14-a553-47be-9bd8-a5de527a88ca-istiod-ca-cert\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.129770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129681 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92939b14-a553-47be-9bd8-a5de527a88ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92939b14-a553-47be-9bd8-a5de527a88ca-istio-podinfo\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.129770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129767 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-kube-api-access-86cqz\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-kube-api-access-86cqz\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.129872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129797 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-istio-token\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-istio-token\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.129872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129826 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-envoy\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.129872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129862 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-data\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.129962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129886 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-certs\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.129962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.129913 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-credential-socket\") pod \"92939b14-a553-47be-9bd8-a5de527a88ca\" (UID: \"92939b14-a553-47be-9bd8-a5de527a88ca\") " Nov 02 14:12:16.130793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.130672 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-certs" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:16.130793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.130770 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-credential-socket" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:16.130972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.130793 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92939b14-a553-47be-9bd8-a5de527a88ca-istiod-ca-cert" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:16.130972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.130882 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-data" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:16.130972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.130917 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-socket" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:16.159483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.158337 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-envoy" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:16.159483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.159265 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-kube-api-access-86cqz" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "kube-api-access-86cqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:16.160264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.160231 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-istio-token" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:16.160392 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.160327 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92939b14-a553-47be-9bd8-a5de527a88ca-istio-podinfo" pod "92939b14-a553-47be-9bd8-a5de527a88ca" (UID: "92939b14-a553-47be-9bd8-a5de527a88ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:12:16.230944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.230909 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.230944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.230940 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.230944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.230950 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.231325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.230959 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.231325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.230968 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92939b14-a553-47be-9bd8-a5de527a88ca-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.231325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.230976 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92939b14-a553-47be-9bd8-a5de527a88ca-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.231325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.231022 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86cqz\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-kube-api-access-86cqz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.231325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.231035 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92939b14-a553-47be-9bd8-a5de527a88ca-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.231325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.231049 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92939b14-a553-47be-9bd8-a5de527a88ca-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:16.373566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.373473 5591 generic.go:356] "Generic (PLEG): container finished" podID="92939b14-a553-47be-9bd8-a5de527a88ca" containerID="b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740" exitCode=0 Nov 02 14:12:16.373566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.373517 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" Nov 02 14:12:16.373807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.373657 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" event={"ID":"92939b14-a553-47be-9bd8-a5de527a88ca","Type":"ContainerDied","Data":"b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740"} Nov 02 14:12:16.373807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.373741 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2" event={"ID":"92939b14-a553-47be-9bd8-a5de527a88ca","Type":"ContainerDied","Data":"8f09a71993f240eff3a2be759f2caaa69bde0a1d99ecab30b99c1f7ebc99e990"} Nov 02 14:12:16.373807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.373774 5591 scope.go:122] "RemoveContainer" containerID="b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740" Nov 02 14:12:16.384574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.384554 5591 scope.go:122] "RemoveContainer" containerID="030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e" Nov 02 14:12:16.397236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.397179 5591 scope.go:122] "RemoveContainer" containerID="b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740" Nov 02 14:12:16.397621 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:16.397602 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740\": container with ID starting with b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740 not found: ID does not exist" containerID="b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740" Nov 02 14:12:16.397701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.397631 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740"} err="failed to get container status \"b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740\": rpc error: code = NotFound desc = could not find container \"b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740\": container with ID starting with b849be3b443f4f210b674e33db58f9bd4bc2782eed1b4704ed5e3a7e95b00740 not found: ID does not exist" Nov 02 14:12:16.397701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.397648 5591 scope.go:122] "RemoveContainer" containerID="030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e" Nov 02 14:12:16.397874 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:16.397855 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e\": container with ID starting with 030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e not found: ID does not exist" containerID="030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e" Nov 02 14:12:16.397930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.397881 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e"} err="failed to get container status \"030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e\": rpc error: code = NotFound desc = could not find container \"030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e\": container with ID starting with 030b9f19d3268540ed231f7d43e8d9765cdfb8a05c535f29c2ea80da4e9b270e not found: ID does not exist" Nov 02 14:12:16.456253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.455410 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:16.459638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.459606 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2"] Nov 02 14:12:16.462756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.462728 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-de493d-openshift-default-5b6d6fc87-9zvn2"] Nov 02 14:12:16.674252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.674214 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:16.854007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.843429 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:12:16.854007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:16.843505 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:12:17.477173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:17.477120 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" path="/var/lib/kubelet/pods/92939b14-a553-47be-9bd8-a5de527a88ca/volumes" Nov 02 14:12:17.487588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:17.487531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:17.852068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:17.843073 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:12:17.852068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:17.843136 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:12:18.212493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.212463 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:18.272101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272056 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn"] Nov 02 14:12:18.272646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272632 5591 cpu_manager.go:414] "containerMap: removing container" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" Nov 02 14:12:18.272691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272647 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" Nov 02 14:12:18.272691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272684 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" Nov 02 14:12:18.272691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272691 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" Nov 02 14:12:18.272807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272699 5591 cpu_manager.go:414] "containerMap: removing container" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" Nov 02 14:12:18.272807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272704 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82deb4cb-c2b7-4454-82d4-51a9f3959707" containerName="istio-proxy" Nov 02 14:12:18.272807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272785 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" Nov 02 14:12:18.272807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272800 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9b8850-3749-47d0-b47e-f77f649f77d3" containerName="istio-proxy" Nov 02 14:12:18.272974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.272818 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c868206f-cc9a-4ba2-bcd0-f4f2fb8e03bc" containerName="istio-proxy" Nov 02 14:12:18.277719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.277701 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:18.295795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.295770 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn"] Nov 02 14:12:18.306645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.306582 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c07f7e\"" Nov 02 14:12:18.362021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.361944 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7"] Nov 02 14:12:18.362662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.362637 5591 cpu_manager.go:414] "containerMap: removing container" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" Nov 02 14:12:18.362662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.362663 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" Nov 02 14:12:18.363265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.363241 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="92939b14-a553-47be-9bd8-a5de527a88ca" containerName="istio-proxy" Nov 02 14:12:18.366777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.366757 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:18.372022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.372003 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7"] Nov 02 14:12:18.387035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.387000 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c07f7e-dockercfg-fpf9z\"" Nov 02 14:12:18.390815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.390775 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1c5d-c247-4a87-9e21-36c191d20169-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn\" (UID: \"2e4d1c5d-c247-4a87-9e21-36c191d20169\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:18.391201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.391182 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96rf2\" (UniqueName: \"kubernetes.io/projected/2e4d1c5d-c247-4a87-9e21-36c191d20169-kube-api-access-96rf2\") pod \"payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn\" (UID: \"2e4d1c5d-c247-4a87-9e21-36c191d20169\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:18.444621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.444472 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:18.561065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.547817 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx9r7\" (UniqueName: \"kubernetes.io/projected/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-kube-api-access-xx9r7\") pod \"payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7\" (UID: \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:18.561065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.547886 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1c5d-c247-4a87-9e21-36c191d20169-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn\" (UID: \"2e4d1c5d-c247-4a87-9e21-36c191d20169\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:18.561065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.548100 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7\" (UID: \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:18.561065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.548239 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96rf2\" (UniqueName: \"kubernetes.io/projected/2e4d1c5d-c247-4a87-9e21-36c191d20169-kube-api-access-96rf2\") pod \"payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn\" (UID: \"2e4d1c5d-c247-4a87-9e21-36c191d20169\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:18.561065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.549728 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1c5d-c247-4a87-9e21-36c191d20169-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn\" (UID: \"2e4d1c5d-c247-4a87-9e21-36c191d20169\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:18.593888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.593806 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96rf2\" (UniqueName: \"kubernetes.io/projected/2e4d1c5d-c247-4a87-9e21-36c191d20169-kube-api-access-96rf2\") pod \"payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn\" (UID: \"2e4d1c5d-c247-4a87-9e21-36c191d20169\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:18.656069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.655048 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7\" (UID: \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:18.656069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.655225 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx9r7\" (UniqueName: \"kubernetes.io/projected/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-kube-api-access-xx9r7\") pod \"payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7\" (UID: \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:18.656508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.656480 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7\" (UID: \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:18.680548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.680509 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx9r7\" (UniqueName: \"kubernetes.io/projected/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-kube-api-access-xx9r7\") pod \"payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7\" (UID: \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:18.696264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.696234 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:18.857346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.841378 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:12:18.857346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.841439 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:12:18.891711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:18.891681 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:19.061182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.061139 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7"] Nov 02 14:12:19.062390 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:12:19.062354 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf76be7d_09c4_45a1_b2a4_4cf9a2db31e6.slice/crio-6fdc7606d9f39825230f834ace55c1e18c9ca2c2952abfb5f53fee7ff1b1802c WatchSource:0}: Error finding container 6fdc7606d9f39825230f834ace55c1e18c9ca2c2952abfb5f53fee7ff1b1802c: Status 404 returned error can't find the container with id 6fdc7606d9f39825230f834ace55c1e18c9ca2c2952abfb5f53fee7ff1b1802c Nov 02 14:12:19.110757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.110700 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:19.218368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.218338 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn"] Nov 02 14:12:19.219562 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:12:19.219534 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4d1c5d_c247_4a87_9e21_36c191d20169.slice/crio-ff258583766d268749001b8ac4417d8d747e498217746442d472100f098ee08a WatchSource:0}: Error finding container ff258583766d268749001b8ac4417d8d747e498217746442d472100f098ee08a: Status 404 returned error can't find the container with id ff258583766d268749001b8ac4417d8d747e498217746442d472100f098ee08a Nov 02 14:12:19.404253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.404219 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" event={"ID":"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6","Type":"ContainerStarted","Data":"e8b0f4a4d525f3cf2bf9123345ec3a663f348b21fa19c2bc1d30a7308090306c"} Nov 02 14:12:19.404457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.404268 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" event={"ID":"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6","Type":"ContainerStarted","Data":"6fdc7606d9f39825230f834ace55c1e18c9ca2c2952abfb5f53fee7ff1b1802c"} Nov 02 14:12:19.404457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.404341 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:19.405817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.405794 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" event={"ID":"2e4d1c5d-c247-4a87-9e21-36c191d20169","Type":"ContainerStarted","Data":"edb972662e3449087723ccb7bccb449a1f213d2c93398abfec0177dc758a1d11"} Nov 02 14:12:19.405905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.405828 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" event={"ID":"2e4d1c5d-c247-4a87-9e21-36c191d20169","Type":"ContainerStarted","Data":"ff258583766d268749001b8ac4417d8d747e498217746442d472100f098ee08a"} Nov 02 14:12:19.405905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.405867 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:19.450022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.449135 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" podStartSLOduration=1.449114622 podStartE2EDuration="1.449114622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:19.448599323 +0000 UTC m=+1688.174221094" watchObservedRunningTime="2026-11-02 14:12:19.449114622 +0000 UTC m=+1688.174736394" Nov 02 14:12:19.491427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.491374 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" podStartSLOduration=1.4913597539999999 podStartE2EDuration="1.491359754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:19.491011198 +0000 UTC m=+1688.216632962" watchObservedRunningTime="2026-11-02 14:12:19.491359754 +0000 UTC m=+1688.216981523" Nov 02 14:12:19.744975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.744885 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:19.833029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.832979 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:12:19.833189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:19.833069 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:12:21.125569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:21.125534 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:21.126087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:21.125846 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:21.126666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:21.126646 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:12:21.588404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:21.588359 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:12:21.670994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:21.670941 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:21.884299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:21.884255 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:22.073006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:22.068399 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:23.111323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:23.111286 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:23.583004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:23.582954 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:24.906398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:24.906368 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:25.366067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:25.364002 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:27.075561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:27.072215 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:28.649408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:28.649038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:29.809961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:29.809927 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd"] Nov 02 14:12:30.060498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.060415 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:30.063676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.063646 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh"] Nov 02 14:12:30.064287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.064230 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" podUID="816d3fdb-8e14-4eda-b40d-3a4da1499c91" containerName="maas-api" containerID="cri-o://21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671" gracePeriod=30 Nov 02 14:12:30.228935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.228897 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b"] Nov 02 14:12:30.229336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.229177 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" podUID="122ac438-de00-431f-829b-a2d925799aa2" containerName="payload-processing" containerID="cri-o://d60a6abf62f15a21b083d5108b4aee7410560aaa0be1e922ff0e8317fc269f5d" gracePeriod=30 Nov 02 14:12:30.234549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.234516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:30.257244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.257220 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm"] Nov 02 14:12:30.257588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.257561 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" podUID="a9417156-daa1-4147-86df-f8fbadca2ac9" containerName="payload-pre-processing" containerID="cri-o://3b59a0ae99f3dafe76fd77c45a0415b632ee5d67ace036ab864a6aa9d5a4a7c3" gracePeriod=30 Nov 02 14:12:30.390654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.390623 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:12:30.397864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.397787 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:30.418318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.418012 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:12:30.418318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.418153 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:12:30.431023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.430932 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816d3fdb-8e14-4eda-b40d-3a4da1499c91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d3fdb-8e14-4eda-b40d-3a4da1499c91-openshift-service-ca\") pod \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " Nov 02 14:12:30.431223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.431040 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816d3fdb-8e14-4eda-b40d-3a4da1499c91-kube-api-access-hmpfw\" (UniqueName: \"kubernetes.io/projected/816d3fdb-8e14-4eda-b40d-3a4da1499c91-kube-api-access-hmpfw\") pod \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " Nov 02 14:12:30.431223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.431150 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-metrics-certs\") pod \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " Nov 02 14:12:30.431357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.431226 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-maas-api-tls\") pod \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\" (UID: \"816d3fdb-8e14-4eda-b40d-3a4da1499c91\") " Nov 02 14:12:30.431647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.431617 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816d3fdb-8e14-4eda-b40d-3a4da1499c91-openshift-service-ca" pod "816d3fdb-8e14-4eda-b40d-3a4da1499c91" (UID: "816d3fdb-8e14-4eda-b40d-3a4da1499c91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:30.431771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.431751 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d3fdb-8e14-4eda-b40d-3a4da1499c91-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:30.436863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.436682 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-maas-api-tls" pod "816d3fdb-8e14-4eda-b40d-3a4da1499c91" (UID: "816d3fdb-8e14-4eda-b40d-3a4da1499c91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:30.436863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.436713 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816d3fdb-8e14-4eda-b40d-3a4da1499c91-kube-api-access-hmpfw" pod "816d3fdb-8e14-4eda-b40d-3a4da1499c91" (UID: "816d3fdb-8e14-4eda-b40d-3a4da1499c91"). InnerVolumeSpecName "kube-api-access-hmpfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:30.438359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.438148 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-metrics-certs" pod "816d3fdb-8e14-4eda-b40d-3a4da1499c91" (UID: "816d3fdb-8e14-4eda-b40d-3a4da1499c91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:30.499078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.499048 5591 generic.go:356] "Generic (PLEG): container finished" podID="a9417156-daa1-4147-86df-f8fbadca2ac9" containerID="3b59a0ae99f3dafe76fd77c45a0415b632ee5d67ace036ab864a6aa9d5a4a7c3" exitCode=0 Nov 02 14:12:30.499299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.499115 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" event={"ID":"a9417156-daa1-4147-86df-f8fbadca2ac9","Type":"ContainerDied","Data":"3b59a0ae99f3dafe76fd77c45a0415b632ee5d67ace036ab864a6aa9d5a4a7c3"} Nov 02 14:12:30.501064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.501034 5591 generic.go:356] "Generic (PLEG): container finished" podID="816d3fdb-8e14-4eda-b40d-3a4da1499c91" containerID="21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671" exitCode=0 Nov 02 14:12:30.501244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.501181 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" event={"ID":"816d3fdb-8e14-4eda-b40d-3a4da1499c91","Type":"ContainerDied","Data":"21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671"} Nov 02 14:12:30.501244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.501196 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" Nov 02 14:12:30.501244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.501237 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh" event={"ID":"816d3fdb-8e14-4eda-b40d-3a4da1499c91","Type":"ContainerDied","Data":"fe3645a7549a8b0f797de52105b74e338337ee99fbd28673cd8a4b3b17ab5ca0"} Nov 02 14:12:30.501468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.501267 5591 scope.go:122] "RemoveContainer" containerID="21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671" Nov 02 14:12:30.503624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.503539 5591 generic.go:356] "Generic (PLEG): container finished" podID="122ac438-de00-431f-829b-a2d925799aa2" containerID="d60a6abf62f15a21b083d5108b4aee7410560aaa0be1e922ff0e8317fc269f5d" exitCode=0 Nov 02 14:12:30.503731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.503643 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" event={"ID":"122ac438-de00-431f-829b-a2d925799aa2","Type":"ContainerDied","Data":"d60a6abf62f15a21b083d5108b4aee7410560aaa0be1e922ff0e8317fc269f5d"} Nov 02 14:12:30.520144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.520118 5591 scope.go:122] "RemoveContainer" containerID="21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671" Nov 02 14:12:30.520696 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:30.520650 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671\": container with ID starting with 21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671 not found: ID does not exist" containerID="21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671" Nov 02 14:12:30.520696 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.520686 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671"} err="failed to get container status \"21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671\": rpc error: code = NotFound desc = could not find container \"21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671\": container with ID starting with 21e84a4722712e424dd0321846fdc3f2f2079ae051f20fb79f16795431850671 not found: ID does not exist" Nov 02 14:12:30.534905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.534879 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:12:30.536582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.536557 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:30.536754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.536739 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d3fdb-8e14-4eda-b40d-3a4da1499c91-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:30.536858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.536846 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmpfw\" (UniqueName: \"kubernetes.io/projected/816d3fdb-8e14-4eda-b40d-3a4da1499c91-kube-api-access-hmpfw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:30.562766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.562690 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh"] Nov 02 14:12:30.563734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.563709 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:12:30.565189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.565165 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f2fe-bb5fb59f5-p87xh"] Nov 02 14:12:30.708163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.694638 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122ac438-de00-431f-829b-a2d925799aa2-kube-api-access-kt8rm\" (UniqueName: \"kubernetes.io/projected/122ac438-de00-431f-829b-a2d925799aa2-kube-api-access-kt8rm\") pod \"122ac438-de00-431f-829b-a2d925799aa2\" (UID: \"122ac438-de00-431f-829b-a2d925799aa2\") " Nov 02 14:12:30.708635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.708601 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9417156-daa1-4147-86df-f8fbadca2ac9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9417156-daa1-4147-86df-f8fbadca2ac9-plugins-config-volume\") pod \"a9417156-daa1-4147-86df-f8fbadca2ac9\" (UID: \"a9417156-daa1-4147-86df-f8fbadca2ac9\") " Nov 02 14:12:30.708891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.708869 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9417156-daa1-4147-86df-f8fbadca2ac9-kube-api-access-pbpkf\" (UniqueName: \"kubernetes.io/projected/a9417156-daa1-4147-86df-f8fbadca2ac9-kube-api-access-pbpkf\") pod \"a9417156-daa1-4147-86df-f8fbadca2ac9\" (UID: \"a9417156-daa1-4147-86df-f8fbadca2ac9\") " Nov 02 14:12:30.709063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.709041 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122ac438-de00-431f-829b-a2d925799aa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122ac438-de00-431f-829b-a2d925799aa2-plugins-config-volume\") pod \"122ac438-de00-431f-829b-a2d925799aa2\" (UID: \"122ac438-de00-431f-829b-a2d925799aa2\") " Nov 02 14:12:30.720514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.719261 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9417156-daa1-4147-86df-f8fbadca2ac9-kube-api-access-pbpkf" pod "a9417156-daa1-4147-86df-f8fbadca2ac9" (UID: "a9417156-daa1-4147-86df-f8fbadca2ac9"). InnerVolumeSpecName "kube-api-access-pbpkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:30.720514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.720465 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9417156-daa1-4147-86df-f8fbadca2ac9-plugins-config-volume" pod "a9417156-daa1-4147-86df-f8fbadca2ac9" (UID: "a9417156-daa1-4147-86df-f8fbadca2ac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:30.722099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.722057 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122ac438-de00-431f-829b-a2d925799aa2-kube-api-access-kt8rm" pod "122ac438-de00-431f-829b-a2d925799aa2" (UID: "122ac438-de00-431f-829b-a2d925799aa2"). InnerVolumeSpecName "kube-api-access-kt8rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:30.727244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.727214 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122ac438-de00-431f-829b-a2d925799aa2-plugins-config-volume" pod "122ac438-de00-431f-829b-a2d925799aa2" (UID: "122ac438-de00-431f-829b-a2d925799aa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:30.810884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.810840 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9417156-daa1-4147-86df-f8fbadca2ac9-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:30.810884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.810877 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbpkf\" (UniqueName: \"kubernetes.io/projected/a9417156-daa1-4147-86df-f8fbadca2ac9-kube-api-access-pbpkf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:30.810884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.810890 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122ac438-de00-431f-829b-a2d925799aa2-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:30.811548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.810899 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8rm\" (UniqueName: \"kubernetes.io/projected/122ac438-de00-431f-829b-a2d925799aa2-kube-api-access-kt8rm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:30.911067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:30.908096 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:31.074437 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:31.074400 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:12:31.431272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.431238 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816d3fdb-8e14-4eda-b40d-3a4da1499c91" path="/var/lib/kubelet/pods/816d3fdb-8e14-4eda-b40d-3a4da1499c91/volumes" Nov 02 14:12:31.511008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.510958 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" Nov 02 14:12:31.511163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.511007 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b" event={"ID":"122ac438-de00-431f-829b-a2d925799aa2","Type":"ContainerDied","Data":"7124f377348e5cc6b839aae8191aa0207c28f8dfc62befacb04c0df6fed1aa3d"} Nov 02 14:12:31.511163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.511064 5591 scope.go:122] "RemoveContainer" containerID="d60a6abf62f15a21b083d5108b4aee7410560aaa0be1e922ff0e8317fc269f5d" Nov 02 14:12:31.512481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.512461 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" event={"ID":"a9417156-daa1-4147-86df-f8fbadca2ac9","Type":"ContainerDied","Data":"04828ad8426886d3b6808625049061cebedef79304a03d50e177ff9c2262130c"} Nov 02 14:12:31.512548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.512501 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm" Nov 02 14:12:31.522252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.522233 5591 scope.go:122] "RemoveContainer" containerID="3b59a0ae99f3dafe76fd77c45a0415b632ee5d67ace036ab864a6aa9d5a4a7c3" Nov 02 14:12:31.575041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.575004 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b"] Nov 02 14:12:31.587716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.587686 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f2fe-7c97f47cf6-mtf9b"] Nov 02 14:12:31.614605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.614567 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm"] Nov 02 14:12:31.616770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:31.616749 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-54f2fe-5d8f444487-2snnm"] Nov 02 14:12:32.386385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.386352 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw"] Nov 02 14:12:32.387139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387122 5591 cpu_manager.go:414] "containerMap: removing container" podUID="816d3fdb-8e14-4eda-b40d-3a4da1499c91" containerName="maas-api" Nov 02 14:12:32.387204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387142 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816d3fdb-8e14-4eda-b40d-3a4da1499c91" containerName="maas-api" Nov 02 14:12:32.387204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387161 5591 cpu_manager.go:414] "containerMap: removing container" podUID="a9417156-daa1-4147-86df-f8fbadca2ac9" containerName="payload-pre-processing" Nov 02 14:12:32.387204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387170 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9417156-daa1-4147-86df-f8fbadca2ac9" containerName="payload-pre-processing" Nov 02 14:12:32.387301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387208 5591 cpu_manager.go:414] "containerMap: removing container" podUID="122ac438-de00-431f-829b-a2d925799aa2" containerName="payload-processing" Nov 02 14:12:32.387301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387216 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122ac438-de00-431f-829b-a2d925799aa2" containerName="payload-processing" Nov 02 14:12:32.387367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387333 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="816d3fdb-8e14-4eda-b40d-3a4da1499c91" containerName="maas-api" Nov 02 14:12:32.387367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387347 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9417156-daa1-4147-86df-f8fbadca2ac9" containerName="payload-pre-processing" Nov 02 14:12:32.387367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.387362 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="122ac438-de00-431f-829b-a2d925799aa2" containerName="payload-processing" Nov 02 14:12:32.392634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.392609 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.424420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.424390 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c07f7e-serving-cert\"" Nov 02 14:12:32.433023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.429434 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw"] Nov 02 14:12:32.526963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.526920 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:32.531134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.531101 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.531300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.531194 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-metrics-certs\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.531300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.531281 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cqw\" (UniqueName: \"kubernetes.io/projected/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-kube-api-access-p2cqw\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.531431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.531412 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.655146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.655047 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cqw\" (UniqueName: \"kubernetes.io/projected/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-kube-api-access-p2cqw\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.683153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.683110 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.683557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.683534 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.683732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.683718 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-metrics-certs\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.696073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.685099 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.696073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.693580 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-metrics-certs\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.696610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.696575 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.697101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.697076 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cqw\" (UniqueName: \"kubernetes.io/projected/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-kube-api-access-p2cqw\") pod \"maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.707649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.707623 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:32.740173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:32.740125 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:33.021384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:33.021345 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw"] Nov 02 14:12:33.022769 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:12:33.022741 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32c7ac4_b4a0_41a4_b14a_6de00fed3263.slice/crio-b5bf263e51f4c0047e34525b503f9f7f53660e8f31f38d40333b96e79e715efe WatchSource:0}: Error finding container b5bf263e51f4c0047e34525b503f9f7f53660e8f31f38d40333b96e79e715efe: Status 404 returned error can't find the container with id b5bf263e51f4c0047e34525b503f9f7f53660e8f31f38d40333b96e79e715efe Nov 02 14:12:33.024550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:33.024526 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:12:33.432524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:33.432493 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122ac438-de00-431f-829b-a2d925799aa2" path="/var/lib/kubelet/pods/122ac438-de00-431f-829b-a2d925799aa2/volumes" Nov 02 14:12:33.432934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:33.432920 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9417156-daa1-4147-86df-f8fbadca2ac9" path="/var/lib/kubelet/pods/a9417156-daa1-4147-86df-f8fbadca2ac9/volumes" Nov 02 14:12:33.573725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:33.573689 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" event={"ID":"b32c7ac4-b4a0-41a4-b14a-6de00fed3263","Type":"ContainerStarted","Data":"b5bf263e51f4c0047e34525b503f9f7f53660e8f31f38d40333b96e79e715efe"} Nov 02 14:12:34.286619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:34.285876 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:34.581550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:34.581462 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" event={"ID":"b32c7ac4-b4a0-41a4-b14a-6de00fed3263","Type":"ContainerStarted","Data":"4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088"} Nov 02 14:12:34.581935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:34.581579 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:34.626896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:34.626846 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" podStartSLOduration=2.204468333 podStartE2EDuration="2.626832771s" totalImagesPullingTime="422.364438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:34.625084539 +0000 UTC m=+1703.350706305" watchObservedRunningTime="2026-11-02 14:12:34.626832771 +0000 UTC m=+1703.352454541" Nov 02 14:12:35.037358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:35.037321 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:35.208057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:35.208021 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:35.639566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:35.639525 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:35.925187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:35.925098 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq"] Nov 02 14:12:35.925447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:35.925413 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" containerID="cri-o://eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5" gracePeriod=30 Nov 02 14:12:36.005077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:36.005040 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:36.418381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:36.406637 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:38.170010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:38.160587 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:39.962128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:39.962091 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:40.599718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:40.599682 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:12:41.176396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.176367 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:12:41.245163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245078 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-credential-socket\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245179 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-envoy\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245222 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea174eb3-9111-4203-ac02-d7156399a40f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea174eb3-9111-4203-ac02-d7156399a40f-istiod-ca-cert\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245299 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-kube-api-access-xtj4d\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-kube-api-access-xtj4d\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245342 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-certs\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245387 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-credential-socket" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:41.245500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245431 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea174eb3-9111-4203-ac02-d7156399a40f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea174eb3-9111-4203-ac02-d7156399a40f-istio-podinfo\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245483 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-data\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245524 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-socket\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245555 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-istio-token\") pod \"ea174eb3-9111-4203-ac02-d7156399a40f\" (UID: \"ea174eb3-9111-4203-ac02-d7156399a40f\") " Nov 02 14:12:41.245724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245680 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-certs" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:41.245724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245702 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea174eb3-9111-4203-ac02-d7156399a40f-istiod-ca-cert" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:41.245935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245845 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-data" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:41.245935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.245881 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-socket" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:41.246207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.246180 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.246392 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.246367 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.246501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.246395 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.246501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.246411 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea174eb3-9111-4203-ac02-d7156399a40f-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.246501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.246425 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.248080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.248049 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea174eb3-9111-4203-ac02-d7156399a40f-istio-podinfo" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:12:41.248194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.248158 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-istio-token" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:41.248194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.248189 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-kube-api-access-xtj4d" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "kube-api-access-xtj4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:41.248411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.248395 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-envoy" pod "ea174eb3-9111-4203-ac02-d7156399a40f" (UID: "ea174eb3-9111-4203-ac02-d7156399a40f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:41.347866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.347831 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea174eb3-9111-4203-ac02-d7156399a40f-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.347866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.347861 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtj4d\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-kube-api-access-xtj4d\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.348115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.347876 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea174eb3-9111-4203-ac02-d7156399a40f-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.348115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.347888 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea174eb3-9111-4203-ac02-d7156399a40f-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:41.658364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.656742 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:41.689338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.689300 5591 generic.go:356] "Generic (PLEG): container finished" podID="ea174eb3-9111-4203-ac02-d7156399a40f" containerID="eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5" exitCode=0 Nov 02 14:12:41.689620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.689369 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" Nov 02 14:12:41.689742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.689392 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" event={"ID":"ea174eb3-9111-4203-ac02-d7156399a40f","Type":"ContainerDied","Data":"eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5"} Nov 02 14:12:41.689742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.689718 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq" event={"ID":"ea174eb3-9111-4203-ac02-d7156399a40f","Type":"ContainerDied","Data":"f5072b705ce74b35ab76102b96a4a1a2ac5cc73bdaec9d83dce6b5f98e719ad7"} Nov 02 14:12:41.689854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.689746 5591 scope.go:122] "RemoveContainer" containerID="eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5" Nov 02 14:12:41.699554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.699530 5591 scope.go:122] "RemoveContainer" containerID="69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd" Nov 02 14:12:41.708857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.708838 5591 scope.go:122] "RemoveContainer" containerID="eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5" Nov 02 14:12:41.709138 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:41.709118 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5\": container with ID starting with eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5 not found: ID does not exist" containerID="eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5" Nov 02 14:12:41.709194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.709150 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5"} err="failed to get container status \"eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5\": rpc error: code = NotFound desc = could not find container \"eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5\": container with ID starting with eddfb12e6d08d5fa8b647d36e599fd850309d9e3bf5e227aaf58711002539cb5 not found: ID does not exist" Nov 02 14:12:41.709194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.709170 5591 scope.go:122] "RemoveContainer" containerID="69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd" Nov 02 14:12:41.709419 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:41.709401 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd\": container with ID starting with 69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd not found: ID does not exist" containerID="69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd" Nov 02 14:12:41.709459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.709425 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd"} err="failed to get container status \"69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd\": rpc error: code = NotFound desc = could not find container \"69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd\": container with ID starting with 69e0ee85e131c0fa62d7f64628e52a875153ff783594a53704b60dde28762cfd not found: ID does not exist" Nov 02 14:12:41.744885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.744850 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq"] Nov 02 14:12:41.757927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:41.757901 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f2fe-openshift-default-68f459bb7c-b42pq"] Nov 02 14:12:43.313040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:43.312979 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:43.431073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:43.431034 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" path="/var/lib/kubelet/pods/ea174eb3-9111-4203-ac02-d7156399a40f/volumes" Nov 02 14:12:44.831135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:44.831056 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" podUID="e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" containerName="main" containerID="cri-o://6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50" gracePeriod=45 Nov 02 14:12:44.888416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:44.888373 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" podUID="e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": dial tcp 10.133.0.140:8000: connect: connection refused" Nov 02 14:12:45.145289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.145201 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:45.185945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.185915 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:12:45.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.368360 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-dshm\") pod \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " Nov 02 14:12:45.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.368496 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-model-cache\") pod \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " Nov 02 14:12:45.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.368584 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tls-certs\") pod \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " Nov 02 14:12:45.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.368639 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-kube-api-access-scxqz\" (UniqueName: \"kubernetes.io/projected/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-kube-api-access-scxqz\") pod \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " Nov 02 14:12:45.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.368704 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tmp-dir\") pod \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " Nov 02 14:12:45.376006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.368765 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-home\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-home\") pod \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\" (UID: \"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2\") " Nov 02 14:12:45.395243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.386891 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-model-cache" pod "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" (UID: "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:45.395243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.387311 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tmp-dir" pod "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" (UID: "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:45.404049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.402346 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tls-certs" pod "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" (UID: "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:45.410020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.409194 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-home" pod "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" (UID: "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:45.411902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.411443 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-dshm" pod "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" (UID: "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:12:45.417033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.413138 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-kube-api-access-scxqz" pod "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" (UID: "e9d9d6d4-dfde-46f4-a41a-c42b19255ed2"). InnerVolumeSpecName "kube-api-access-scxqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:45.480007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.479940 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:45.480007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.479997 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:45.480007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.480014 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:45.480284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.480028 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:45.480284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.480042 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxqz\" (UniqueName: \"kubernetes.io/projected/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-kube-api-access-scxqz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:45.480284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.480057 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:12:45.811078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.808308 5591 generic.go:356] "Generic (PLEG): container finished" podID="e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" containerID="6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50" exitCode=0 Nov 02 14:12:45.811078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.808498 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" event={"ID":"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2","Type":"ContainerDied","Data":"6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50"} Nov 02 14:12:45.811078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.808557 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" event={"ID":"e9d9d6d4-dfde-46f4-a41a-c42b19255ed2","Type":"ContainerDied","Data":"df8219c5b45d274f4895de8e6515b8f194b902baff6cede5b7d7ed676bbc6973"} Nov 02 14:12:45.811078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.808596 5591 scope.go:122] "RemoveContainer" containerID="6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50" Nov 02 14:12:45.811078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.808776 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd" Nov 02 14:12:45.821308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.821293 5591 scope.go:122] "RemoveContainer" containerID="6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50" Nov 02 14:12:45.821701 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:12:45.821667 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50\": container with ID starting with 6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50 not found: ID does not exist" containerID="6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50" Nov 02 14:12:45.821764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.821715 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50"} err="failed to get container status \"6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50\": rpc error: code = NotFound desc = could not find container \"6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50\": container with ID starting with 6cb8175f6186c29c90d7d8721a41167492bdae7a33638125f2439fb2c0b3ae50 not found: ID does not exist" Nov 02 14:12:45.884059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.883993 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd"] Nov 02 14:12:45.901346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:45.901308 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-8acd34/praxis-route-8acd34-kserve-545cbd4999-rljvd"] Nov 02 14:12:46.968615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:46.968578 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:47.433790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:47.433755 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" path="/var/lib/kubelet/pods/e9d9d6d4-dfde-46f4-a41a-c42b19255ed2/volumes" Nov 02 14:12:48.819704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:48.819666 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:50.621596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:50.617723 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:51.848207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:51.848173 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:52.499422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:52.499388 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:52.990354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:52.990318 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.353133 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99"] Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.353996 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.354016 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.354046 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.354056 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.354077 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" containerName="main" Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.354086 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" containerName="main" Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.354210 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" Nov 02 14:12:53.355885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.354226 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d9d6d4-dfde-46f4-a41a-c42b19255ed2" containerName="main" Nov 02 14:12:53.359876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.359846 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.376492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.376427 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cdaf64-openshift-default-dockercfg-g8696\"" Nov 02 14:12:53.391836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.391806 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.392124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.392095 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.392221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.392149 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.392221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.392197 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.392342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.392238 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.392342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.392323 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.392449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.392355 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqsz7\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-kube-api-access-sqsz7\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.392503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.392446 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.392560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.392512 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb599d3-ebb0-419d-806d-d4a34ad60032-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.393151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.393120 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99"] Nov 02 14:12:53.492677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.492638 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:53.494931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.494897 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.508006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.495658 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.511206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.511169 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsz7\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-kube-api-access-sqsz7\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.511906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.511878 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.512183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.512162 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb599d3-ebb0-419d-806d-d4a34ad60032-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.512402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.512381 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.512651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.512630 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.512857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.512838 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.518009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.513034 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.521779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.521748 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb599d3-ebb0-419d-806d-d4a34ad60032-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.521904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.499348 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.522776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.495533 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.522776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.522056 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.522776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.522356 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.562008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.538389 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.562008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.552086 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqsz7\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-kube-api-access-sqsz7\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.571009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.562811 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.571009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.568782 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.674541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.674509 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:53.821266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:53.821235 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:54.002375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:54.002345 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99"] Nov 02 14:12:54.003280 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:12:54.003255 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb599d3_ebb0_419d_806d_d4a34ad60032.slice/crio-341b253c11a51a183d7565e2c70b0f9b03cb4928333c73985be8f244b8d7e348 WatchSource:0}: Error finding container 341b253c11a51a183d7565e2c70b0f9b03cb4928333c73985be8f244b8d7e348: Status 404 returned error can't find the container with id 341b253c11a51a183d7565e2c70b0f9b03cb4928333c73985be8f244b8d7e348 Nov 02 14:12:54.005818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:54.005783 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:54.005922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:54.005850 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:54.005922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:54.005880 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:54.183198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:54.183121 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:54.871588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:54.871551 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" event={"ID":"8eb599d3-ebb0-419d-806d-d4a34ad60032","Type":"ContainerStarted","Data":"eea33cfe63e8d74660a27e236eb08ccb338fffddfbc95b9f2739ffc220551771"} Nov 02 14:12:54.871774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:54.871605 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" event={"ID":"8eb599d3-ebb0-419d-806d-d4a34ad60032","Type":"ContainerStarted","Data":"341b253c11a51a183d7565e2c70b0f9b03cb4928333c73985be8f244b8d7e348"} Nov 02 14:12:54.921303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:54.921246 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podStartSLOduration=1.921227117 podStartE2EDuration="1.921227117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:54.917916333 +0000 UTC m=+1723.643538103" watchObservedRunningTime="2026-11-02 14:12:54.921227117 +0000 UTC m=+1723.646848887" Nov 02 14:12:55.675387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:55.675350 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:55.681155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:55.681129 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:55.779374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:55.779340 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:55.876710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:55.876675 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:55.877589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:55.877568 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:12:57.498926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.498894 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:57.640085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.640051 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5"] Nov 02 14:12:57.640924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.640904 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea174eb3-9111-4203-ac02-d7156399a40f" containerName="istio-proxy" Nov 02 14:12:57.644283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.644264 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.668648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.668617 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr"] Nov 02 14:12:57.672404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.672388 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.679440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679410 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.679588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679511 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.679588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679555 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.679678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679628 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.679678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679668 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.679810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679792 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.679860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679843 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nnsc\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-kube-api-access-2nnsc\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.679963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679949 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.680020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.679999 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.694831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.694800 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5"] Nov 02 14:12:57.710230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.710202 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr"] Nov 02 14:12:57.739354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.739316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:57.812517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.788192 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.814369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.814339 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk24z\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-kube-api-access-xk24z\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.814665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.814643 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.814806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.814788 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.814930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.814916 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.815132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815113 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.815277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815257 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.815385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815372 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815493 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815501 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815570 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nnsc\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-kube-api-access-2nnsc\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815652 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815683 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815777 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815837 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815879 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815912 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.815947 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.816112 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.816963 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.817330 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.836349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.830572 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.837379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.837316 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.837478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.805454 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.838076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.838014 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.875828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.875749 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nnsc\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-kube-api-access-2nnsc\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.883586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.883524 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.917196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917155 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk24z\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-kube-api-access-xk24z\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917250 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917279 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917304 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917360 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917388 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917475 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917524 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917554 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917735 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.917976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917950 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.918066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.917963 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.918162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.918115 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.918272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.918253 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.919780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.919756 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.919889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.919873 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.921530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.921508 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:57.941372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.941345 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk24z\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-kube-api-access-xk24z\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.941453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.941381 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:57.959469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.959408 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:57.985220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:57.985191 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:58.288326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.288293 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5"] Nov 02 14:12:58.291620 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:12:58.291591 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cfb3f5_5903_47dc_9837_27c0f18adabb.slice/crio-5d72e4641b76676fdcfef3bfbf3fa82b28f9394ee5536a1dfe233e4f7e1c16e5 WatchSource:0}: Error finding container 5d72e4641b76676fdcfef3bfbf3fa82b28f9394ee5536a1dfe233e4f7e1c16e5: Status 404 returned error can't find the container with id 5d72e4641b76676fdcfef3bfbf3fa82b28f9394ee5536a1dfe233e4f7e1c16e5 Nov 02 14:12:58.294098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.294066 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:58.294168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.294132 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:58.294168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.294161 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:58.311854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.311822 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr"] Nov 02 14:12:58.313196 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:12:58.313165 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d9e0b3_69fc_4536_b3e2_be7e8d6714be.slice/crio-bff2e4daf014ce542cf1c08add58d2056ff2340c2a8ea5b2edcf1edcd976ff53 WatchSource:0}: Error finding container bff2e4daf014ce542cf1c08add58d2056ff2340c2a8ea5b2edcf1edcd976ff53: Status 404 returned error can't find the container with id bff2e4daf014ce542cf1c08add58d2056ff2340c2a8ea5b2edcf1edcd976ff53 Nov 02 14:12:58.315818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.315786 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:58.315918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.315861 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:58.315918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.315896 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:12:58.902850 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.902811 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" event={"ID":"27d9e0b3-69fc-4536-b3e2-be7e8d6714be","Type":"ContainerStarted","Data":"1d9684187d171283d535e2ab15e9f75ad71f2496534120879eb2b8f296694dc6"} Nov 02 14:12:58.903392 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.902901 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" event={"ID":"27d9e0b3-69fc-4536-b3e2-be7e8d6714be","Type":"ContainerStarted","Data":"bff2e4daf014ce542cf1c08add58d2056ff2340c2a8ea5b2edcf1edcd976ff53"} Nov 02 14:12:58.904227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.904200 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" event={"ID":"f3cfb3f5-5903-47dc-9837-27c0f18adabb","Type":"ContainerStarted","Data":"45192053bf7cb9a2e18f0afb88a9a46df1b9b0266f37afa24614e5ac5e3a7a15"} Nov 02 14:12:58.904349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.904247 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" event={"ID":"f3cfb3f5-5903-47dc-9837-27c0f18adabb","Type":"ContainerStarted","Data":"5d72e4641b76676fdcfef3bfbf3fa82b28f9394ee5536a1dfe233e4f7e1c16e5"} Nov 02 14:12:58.963153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.963109 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:12:58.985434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:58.985399 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:12:59.023934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.023883 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" podStartSLOduration=2.023872084 podStartE2EDuration="2.023872084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:59.022964072 +0000 UTC m=+1727.748585842" watchObservedRunningTime="2026-11-02 14:12:59.023872084 +0000 UTC m=+1727.749493853" Nov 02 14:12:59.025290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.025255 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" podStartSLOduration=2.025242094 podStartE2EDuration="2.025242094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:58.976169887 +0000 UTC m=+1727.701791659" watchObservedRunningTime="2026-11-02 14:12:59.025242094 +0000 UTC m=+1727.750863864" Nov 02 14:12:59.289356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.289197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:59.483227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.483193 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:59.875022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.874920 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:12:59.980046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.966030 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 14:12:59.980046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.966126 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 14:12:59.992009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.991476 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 14:12:59.992009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:12:59.991560 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 14:13:00.963010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:00.962380 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 14:13:00.963010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:00.962479 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 14:13:00.985653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:00.985618 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 14:13:00.986107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:00.986073 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 14:13:01.101728 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:13:01.101691 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:13:01.725606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:01.725555 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:01.960491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:01.960451 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 14:13:01.960788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:01.960753 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 14:13:01.985956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:01.985842 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 14:13:01.985956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:01.985916 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 14:13:02.961477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:02.961435 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 14:13:02.961811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:02.961777 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 14:13:02.987386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:02.987349 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 14:13:02.987911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:02.987398 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 14:13:03.614192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:03.614144 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:03.962009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:03.959969 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 14:13:03.962009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:03.960054 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 14:13:03.986497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:03.986466 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 14:13:03.986757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:03.986733 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 14:13:04.960705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:04.960666 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 14:13:04.961155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:04.960742 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 14:13:04.986469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:04.986435 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 14:13:04.986651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:04.986501 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 14:13:05.372172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:05.372079 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:05.977033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:05.976961 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:13:05.978145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:05.978121 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:13:05.979117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:05.979094 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:13:05.990263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:05.990239 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:13:05.990514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:05.990489 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:13:05.991403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:05.991386 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:13:06.982087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:06.982052 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:07.150255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:07.150219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:07.185954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:07.185917 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:13:07.421033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:07.420937 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:07.638744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:07.638712 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:09.176348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:09.176295 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:09.500066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:09.498962 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:11.193240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:11.193194 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:12.683674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.683642 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk"] Nov 02 14:13:12.689144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.689125 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.716914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.716878 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk"] Nov 02 14:13:12.718132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.718109 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cdaf64-serving-cert\"" Nov 02 14:13:12.744805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.744773 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-metrics-certs\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.744968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.744855 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzb4\" (UniqueName: \"kubernetes.io/projected/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-kube-api-access-6gzb4\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.745046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.745023 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.745149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.745133 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.817871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.817820 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:12.846688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.846654 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.846847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.846750 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.846847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.846798 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-metrics-certs\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.846921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.846853 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzb4\" (UniqueName: \"kubernetes.io/projected/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-kube-api-access-6gzb4\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.846921 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:13:12.846908 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-serving-cert: secret "maas-api-e2e-praxis-b-cdaf64-serving-cert" not found Nov 02 14:13:12.847035 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:13:12.846977 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls podName:7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e nodeName:}" failed. No retries permitted until 2026-11-02 14:13:13.346956649 +0000 UTC m=+1742.072578400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls") pod "maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" (UID: "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e") : secret "maas-api-e2e-praxis-b-cdaf64-serving-cert" not found Nov 02 14:13:12.847400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.847377 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.849582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.849558 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-metrics-certs\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.905009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.899048 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzb4\" (UniqueName: \"kubernetes.io/projected/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-kube-api-access-6gzb4\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:12.978583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:12.978506 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:13.145527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:13.145493 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:13.354319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:13.354234 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:13.356714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:13.356693 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:13.607886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:13.607785 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:13.977400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:13.977368 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk"] Nov 02 14:13:13.978589 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:13:13.978561 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eea3a05_b1cf_4c9f_8c37_9b0fff5eef7e.slice/crio-287c60a4fcfe9e65f9443e396dcee518272ac60dfa8d6d5dc66cdc912f2385da WatchSource:0}: Error finding container 287c60a4fcfe9e65f9443e396dcee518272ac60dfa8d6d5dc66cdc912f2385da: Status 404 returned error can't find the container with id 287c60a4fcfe9e65f9443e396dcee518272ac60dfa8d6d5dc66cdc912f2385da Nov 02 14:13:14.116012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:14.115962 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" event={"ID":"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e","Type":"ContainerStarted","Data":"287c60a4fcfe9e65f9443e396dcee518272ac60dfa8d6d5dc66cdc912f2385da"} Nov 02 14:13:14.666003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:14.665103 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:15.122834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:15.122737 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" event={"ID":"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e","Type":"ContainerStarted","Data":"11a25eaa933d2fb417e0a8b97617f5ed77e623065ff25d39b1053acb455f2a66"} Nov 02 14:13:15.122834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:15.122823 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:15.167263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:15.167210 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" podStartSLOduration=2.803616896 podStartE2EDuration="3.167197344s" totalImagesPullingTime="363.580448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:15.165759434 +0000 UTC m=+1743.891381204" watchObservedRunningTime="2026-11-02 14:13:15.167197344 +0000 UTC m=+1743.892819114" Nov 02 14:13:15.357142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:15.357087 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:16.023114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.023077 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:16.401004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.395735 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:16.511293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.511262 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f"] Nov 02 14:13:16.515587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.515560 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.546027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.543123 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f"] Nov 02 14:13:16.601298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601258 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hj7\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-kube-api-access-j9hj7\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.601473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601315 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.601473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601386 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.601473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601440 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.601473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601466 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.601634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601536 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.601634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601579 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.601739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601657 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.601739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.601723 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703105 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703155 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703189 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703246 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703307 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703354 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hj7\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-kube-api-access-j9hj7\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703405 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703438 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703504 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703642 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703732 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703828 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.703936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703921 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.704011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.703928 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.705710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.705688 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.705822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.705746 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.727322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.727296 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.727763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.727740 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hj7\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-kube-api-access-j9hj7\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:16.830911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:16.830876 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:17.159541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:17.159515 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f"] Nov 02 14:13:17.160645 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:13:17.160610 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaf7bd7_7b6d_4660_9d46_b10a2a0ea8c4.slice/crio-36f719ca6c59f41208f66512fc889cd4d912a421d3daa22614176100a03bc0be WatchSource:0}: Error finding container 36f719ca6c59f41208f66512fc889cd4d912a421d3daa22614176100a03bc0be: Status 404 returned error can't find the container with id 36f719ca6c59f41208f66512fc889cd4d912a421d3daa22614176100a03bc0be Nov 02 14:13:17.163108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:17.163071 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:17.163211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:17.163169 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:17.163305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:17.163214 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:18.052429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.052395 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:18.161019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.160277 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" event={"ID":"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4","Type":"ContainerStarted","Data":"b966e1d6eda774d8c3e231bf4359741f320982bc2f1dcc29669383f1b2251454"} Nov 02 14:13:18.161019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.160361 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" event={"ID":"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4","Type":"ContainerStarted","Data":"36f719ca6c59f41208f66512fc889cd4d912a421d3daa22614176100a03bc0be"} Nov 02 14:13:18.220547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.220495 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" podStartSLOduration=2.220481313 podStartE2EDuration="2.220481313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:18.219952534 +0000 UTC m=+1746.945574304" watchObservedRunningTime="2026-11-02 14:13:18.220481313 +0000 UTC m=+1746.946103083" Nov 02 14:13:18.314206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.314100 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:18.686189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.686145 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:18.851007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.844747 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:18.856006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.853096 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 14:13:18.856006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:18.853162 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 14:13:19.856162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:19.856122 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 14:13:19.856685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:19.856572 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 14:13:19.966845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:19.966807 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:20.849747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:20.849714 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 14:13:20.850061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:20.850030 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 14:13:21.156881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:21.156846 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:13:21.800010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:21.792427 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:21.847008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:21.831419 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 14:13:21.847008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:21.831499 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 14:13:22.856008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:22.855147 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 14:13:22.856008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:22.855213 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 14:13:23.025196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:23.025159 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:23.229776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:23.229739 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:23.403738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:23.403704 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:23.832084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:23.832044 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 02 14:13:23.832257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:23.832116 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 02 14:13:24.836290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:24.836260 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:24.836750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:24.836646 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:24.837627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:24.837603 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:13:24.976050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:24.975691 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:26.651411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:26.651365 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:28.386159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:28.386127 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:29.322009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:29.319065 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:30.328406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:30.328347 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:30.521703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:30.521662 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:30.538105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:30.538059 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:30.578269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:30.578220 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:31.087071 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:13:31.087031 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:13:31.912630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:31.912595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:32.759438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:32.759403 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:32.957761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:32.957725 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:33.291935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:33.291900 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:34.501394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:34.501356 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:34.671604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:34.671563 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:36.214817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:36.214753 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:37.423174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:37.423133 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:37.590185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:37.590143 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:37.978887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:37.978850 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:38.181079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:38.181044 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:38.612007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:38.610716 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:39.402288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:39.402253 5591 generic.go:356] "Generic (PLEG): container finished" podID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerID="eea33cfe63e8d74660a27e236eb08ccb338fffddfbc95b9f2739ffc220551771" exitCode=0 Nov 02 14:13:39.402483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:39.402326 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" event={"ID":"8eb599d3-ebb0-419d-806d-d4a34ad60032","Type":"ContainerDied","Data":"eea33cfe63e8d74660a27e236eb08ccb338fffddfbc95b9f2739ffc220551771"} Nov 02 14:13:39.402771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:39.402757 5591 scope.go:122] "RemoveContainer" containerID="eea33cfe63e8d74660a27e236eb08ccb338fffddfbc95b9f2739ffc220551771" Nov 02 14:13:39.404287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:39.404257 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:39.404368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:39.404329 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:39.404368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:39.404362 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:39.486414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:39.486378 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:39.932008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:39.926718 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:40.410117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.410077 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" event={"ID":"8eb599d3-ebb0-419d-806d-d4a34ad60032","Type":"ContainerStarted","Data":"3c46a1091706577a99546d331b68cadd65d823d8313504c7dd78a087d764852e"} Nov 02 14:13:40.466305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.466272 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn"] Nov 02 14:13:40.500825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.500781 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7"] Nov 02 14:13:40.674845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.674748 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:13:40.675936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.675912 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:13:40.676047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.675962 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:13:40.718284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.718249 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:40.877420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.877390 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:13:40.894478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:40.894448 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:41.675059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:41.675025 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:13:41.675473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:41.675078 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:13:41.963716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:41.963618 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:42.351064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:42.349076 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:42.656871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:42.656827 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:13:42.675117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:42.675083 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:13:42.675493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:42.675135 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:13:43.683051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:43.679877 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:43.683051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:43.681843 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:13:43.683051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:43.681896 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:13:44.377009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:44.376696 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b"] Nov 02 14:13:44.741126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:44.740562 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:44.743534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:44.743507 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:13:44.743777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:44.743738 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:13:45.020704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.020614 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:45.381152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.381105 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:45.458007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.457452 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:45.472877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.472833 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" podUID="2e4d1c5d-c247-4a87-9e21-36c191d20169" containerName="payload-processing" containerID="cri-o://edb972662e3449087723ccb7bccb449a1f213d2c93398abfec0177dc758a1d11" gracePeriod=25 Nov 02 14:13:45.506686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.506637 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" podUID="bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6" containerName="payload-pre-processing" containerID="cri-o://e8b0f4a4d525f3cf2bf9123345ec3a663f348b21fa19c2bc1d30a7308090306c" gracePeriod=25 Nov 02 14:13:45.518541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.517771 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:45.518541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.517764 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:45.625030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.619155 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:13:45.692552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.692505 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:13:45.692895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.692848 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:13:45.698024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.693728 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:45.873104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.873059 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:45.973311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:45.967108 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:46.027687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.027107 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:46.027687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.027268 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:46.053126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.051069 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:46.388052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.386375 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:46.388052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.386806 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:13:46.500007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.472238 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:46.500007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.474556 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:13:46.526006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.519114 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:46.526006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.520653 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:13:46.691140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.677724 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:13:46.691140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.678290 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:13:46.691140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.679909 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:13:46.691140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.679949 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:13:46.864918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.852943 5591 generic.go:356] "Generic (PLEG): container finished" podID="bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6" containerID="e8b0f4a4d525f3cf2bf9123345ec3a663f348b21fa19c2bc1d30a7308090306c" exitCode=0 Nov 02 14:13:46.868177 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.865297 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" event={"ID":"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6","Type":"ContainerDied","Data":"e8b0f4a4d525f3cf2bf9123345ec3a663f348b21fa19c2bc1d30a7308090306c"} Nov 02 14:13:46.871544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.871506 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Nov 02 14:13:46.874036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.874007 5591 generic.go:356] "Generic (PLEG): container finished" podID="2e4d1c5d-c247-4a87-9e21-36c191d20169" containerID="edb972662e3449087723ccb7bccb449a1f213d2c93398abfec0177dc758a1d11" exitCode=0 Nov 02 14:13:46.874718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.874693 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" event={"ID":"2e4d1c5d-c247-4a87-9e21-36c191d20169","Type":"ContainerDied","Data":"edb972662e3449087723ccb7bccb449a1f213d2c93398abfec0177dc758a1d11"} Nov 02 14:13:46.906008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.887849 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 14:13:46.906008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.888301 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:13:46.976412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.970784 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:46.976412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:46.971313 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:13:47.056275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.056223 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:47.080222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.068787 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:13:47.087012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.085213 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:47.387166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.387118 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:47.421070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.421036 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:13:47.524079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.474600 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:13:47.524079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.520916 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 14:13:47.526599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.526563 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 02 14:13:47.540653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.540556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:47.599006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.591200 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:13:47.599006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.597692 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:13:47.619020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.617841 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:13:47.619020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.618386 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:13:47.625537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.625505 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-kube-api-access-xx9r7\" (UniqueName: \"kubernetes.io/projected/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-kube-api-access-xx9r7\") pod \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\" (UID: \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\") " Nov 02 14:13:47.626303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.626283 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-plugins-config-volume\") pod \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\" (UID: \"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6\") " Nov 02 14:13:47.629185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.626945 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-plugins-config-volume" pod "bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6" (UID: "bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:47.629320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.629204 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-kube-api-access-xx9r7" pod "bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6" (UID: "bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6"). InnerVolumeSpecName "kube-api-access-xx9r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:47.647211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.647178 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:13:47.649325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.649299 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:13:47.654566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.654545 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:13:47.659355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.659333 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:13:47.677846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.677814 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:13:47.678028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.677877 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:13:47.715206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.715171 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:47.728548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.728510 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4d1c5d-c247-4a87-9e21-36c191d20169-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1c5d-c247-4a87-9e21-36c191d20169-plugins-config-volume\") pod \"2e4d1c5d-c247-4a87-9e21-36c191d20169\" (UID: \"2e4d1c5d-c247-4a87-9e21-36c191d20169\") " Nov 02 14:13:47.728740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.728615 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4d1c5d-c247-4a87-9e21-36c191d20169-kube-api-access-96rf2\" (UniqueName: \"kubernetes.io/projected/2e4d1c5d-c247-4a87-9e21-36c191d20169-kube-api-access-96rf2\") pod \"2e4d1c5d-c247-4a87-9e21-36c191d20169\" (UID: \"2e4d1c5d-c247-4a87-9e21-36c191d20169\") " Nov 02 14:13:47.729199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.729149 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx9r7\" (UniqueName: \"kubernetes.io/projected/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-kube-api-access-xx9r7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:47.729199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.729179 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:47.730409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.730377 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4d1c5d-c247-4a87-9e21-36c191d20169-plugins-config-volume" pod "2e4d1c5d-c247-4a87-9e21-36c191d20169" (UID: "2e4d1c5d-c247-4a87-9e21-36c191d20169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:47.733352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.733323 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4d1c5d-c247-4a87-9e21-36c191d20169-kube-api-access-96rf2" pod "2e4d1c5d-c247-4a87-9e21-36c191d20169" (UID: "2e4d1c5d-c247-4a87-9e21-36c191d20169"). InnerVolumeSpecName "kube-api-access-96rf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:47.830506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.830409 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96rf2\" (UniqueName: \"kubernetes.io/projected/2e4d1c5d-c247-4a87-9e21-36c191d20169-kube-api-access-96rf2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:47.830506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.830452 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1c5d-c247-4a87-9e21-36c191d20169-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:47.885545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.885500 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" event={"ID":"bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6","Type":"ContainerDied","Data":"6fdc7606d9f39825230f834ace55c1e18c9ca2c2952abfb5f53fee7ff1b1802c"} Nov 02 14:13:47.886082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.885573 5591 scope.go:122] "RemoveContainer" containerID="e8b0f4a4d525f3cf2bf9123345ec3a663f348b21fa19c2bc1d30a7308090306c" Nov 02 14:13:47.886082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.885711 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7" Nov 02 14:13:47.889605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.889579 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" Nov 02 14:13:47.894742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.894710 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn" event={"ID":"2e4d1c5d-c247-4a87-9e21-36c191d20169","Type":"ContainerDied","Data":"ff258583766d268749001b8ac4417d8d747e498217746442d472100f098ee08a"} Nov 02 14:13:47.900480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.900456 5591 scope.go:122] "RemoveContainer" containerID="edb972662e3449087723ccb7bccb449a1f213d2c93398abfec0177dc758a1d11" Nov 02 14:13:47.980519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:47.980487 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:48.015303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:48.015273 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn"] Nov 02 14:13:48.019663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:48.019615 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c07f7e-7c5795fb47-m2thn"] Nov 02 14:13:48.058095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:48.058067 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7"] Nov 02 14:13:48.064575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:48.064548 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c07f7e-689859658c-6r6p7"] Nov 02 14:13:48.680705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:48.680673 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:13:48.681713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:48.681691 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:13:49.377257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.377219 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" podUID="0aec9a81-5460-4411-a093-68ac8e84b56d" containerName="payload-processing" containerID="cri-o://fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060" gracePeriod=25 Nov 02 14:13:49.432162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.432129 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4d1c5d-c247-4a87-9e21-36c191d20169" path="/var/lib/kubelet/pods/2e4d1c5d-c247-4a87-9e21-36c191d20169/volumes" Nov 02 14:13:49.432755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.432723 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6" path="/var/lib/kubelet/pods/bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6/volumes" Nov 02 14:13:49.650318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.650293 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:13:49.749830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.749794 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aec9a81-5460-4411-a093-68ac8e84b56d-kube-api-access-hxxh8\" (UniqueName: \"kubernetes.io/projected/0aec9a81-5460-4411-a093-68ac8e84b56d-kube-api-access-hxxh8\") pod \"0aec9a81-5460-4411-a093-68ac8e84b56d\" (UID: \"0aec9a81-5460-4411-a093-68ac8e84b56d\") " Nov 02 14:13:49.750020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.749928 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aec9a81-5460-4411-a093-68ac8e84b56d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aec9a81-5460-4411-a093-68ac8e84b56d-plugins-config-volume\") pod \"0aec9a81-5460-4411-a093-68ac8e84b56d\" (UID: \"0aec9a81-5460-4411-a093-68ac8e84b56d\") " Nov 02 14:13:49.750362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.750330 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aec9a81-5460-4411-a093-68ac8e84b56d-plugins-config-volume" pod "0aec9a81-5460-4411-a093-68ac8e84b56d" (UID: "0aec9a81-5460-4411-a093-68ac8e84b56d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:49.750487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.750476 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aec9a81-5460-4411-a093-68ac8e84b56d-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:49.752004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.751966 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aec9a81-5460-4411-a093-68ac8e84b56d-kube-api-access-hxxh8" pod "0aec9a81-5460-4411-a093-68ac8e84b56d" (UID: "0aec9a81-5460-4411-a093-68ac8e84b56d"). InnerVolumeSpecName "kube-api-access-hxxh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:49.806100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.805584 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:49.852131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.852093 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxxh8\" (UniqueName: \"kubernetes.io/projected/0aec9a81-5460-4411-a093-68ac8e84b56d-kube-api-access-hxxh8\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:49.908286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.908212 5591 generic.go:356] "Generic (PLEG): container finished" podID="0aec9a81-5460-4411-a093-68ac8e84b56d" containerID="fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060" exitCode=0 Nov 02 14:13:49.908431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.908308 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" event={"ID":"0aec9a81-5460-4411-a093-68ac8e84b56d","Type":"ContainerDied","Data":"fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060"} Nov 02 14:13:49.908431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.908333 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" Nov 02 14:13:49.908431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.908348 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b" event={"ID":"0aec9a81-5460-4411-a093-68ac8e84b56d","Type":"ContainerDied","Data":"0e6542edd8a6dad9d6bdc4b8b7723669e261566ddc4bcfe4815906cb80d4d197"} Nov 02 14:13:49.908431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.908370 5591 scope.go:122] "RemoveContainer" containerID="fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060" Nov 02 14:13:49.921520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.921503 5591 scope.go:122] "RemoveContainer" containerID="fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060" Nov 02 14:13:49.921803 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:13:49.921782 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060\": container with ID starting with fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060 not found: ID does not exist" containerID="fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060" Nov 02 14:13:49.921880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.921810 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060"} err="failed to get container status \"fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060\": rpc error: code = NotFound desc = could not find container \"fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060\": container with ID starting with fde3516478aa9d83d2c48ab27a081fa18b1e3ac19b2f099bc5ecd7c1fbd2f060 not found: ID does not exist" Nov 02 14:13:49.962389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.962361 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b"] Nov 02 14:13:49.974111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.974087 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:49.975357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:49.975337 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-732d3a-5555b4df49-6kn4b"] Nov 02 14:13:51.228053 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:51.228017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:51.433313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:51.433270 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aec9a81-5460-4411-a093-68ac8e84b56d" path="/var/lib/kubelet/pods/0aec9a81-5460-4411-a093-68ac8e84b56d/volumes" Nov 02 14:13:52.009272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009235 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878"] Nov 02 14:13:52.009763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009750 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2e4d1c5d-c247-4a87-9e21-36c191d20169" containerName="payload-processing" Nov 02 14:13:52.009810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009764 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4d1c5d-c247-4a87-9e21-36c191d20169" containerName="payload-processing" Nov 02 14:13:52.009810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009785 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0aec9a81-5460-4411-a093-68ac8e84b56d" containerName="payload-processing" Nov 02 14:13:52.009810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009790 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aec9a81-5460-4411-a093-68ac8e84b56d" containerName="payload-processing" Nov 02 14:13:52.009907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009812 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6" containerName="payload-pre-processing" Nov 02 14:13:52.009907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009817 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6" containerName="payload-pre-processing" Nov 02 14:13:52.009971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009912 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf76be7d-09c4-45a1-b2a4-4cf9a2db31e6" containerName="payload-pre-processing" Nov 02 14:13:52.009971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009924 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4d1c5d-c247-4a87-9e21-36c191d20169" containerName="payload-processing" Nov 02 14:13:52.009971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.009933 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aec9a81-5460-4411-a093-68ac8e84b56d" containerName="payload-processing" Nov 02 14:13:52.012947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.012931 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.031770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.031747 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:52.048182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.048156 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878"] Nov 02 14:13:52.103206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103170 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.103405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103213 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.103405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103353 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.103405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103387 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.103544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103411 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.103578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103539 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.103616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103583 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27h4\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-kube-api-access-f27h4\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.103763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103742 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.103803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.103783 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205256 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205344 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205377 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205473 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205508 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205537 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205609 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205660 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27h4\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-kube-api-access-f27h4\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205673 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205756 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.205874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.205858 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.206118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.206096 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.206284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.206260 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.206471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.206445 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.208210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.208182 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.208699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.208678 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.229589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.229561 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.229935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.229911 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27h4\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-kube-api-access-f27h4\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.325641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.325551 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:52.526650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.526612 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:52.649518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.649490 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878"] Nov 02 14:13:52.649799 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:13:52.649776 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac04a52_c885_4b1f_b8c7_c21fe8e3d4bd.slice/crio-041f02e31d4ae07ed580fad58be1dd4920cb2bb1c3fbb915e0533a1c26dd6743 WatchSource:0}: Error finding container 041f02e31d4ae07ed580fad58be1dd4920cb2bb1c3fbb915e0533a1c26dd6743: Status 404 returned error can't find the container with id 041f02e31d4ae07ed580fad58be1dd4920cb2bb1c3fbb915e0533a1c26dd6743 Nov 02 14:13:52.652252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.652220 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:52.652329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.652283 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:52.652329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.652318 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:13:52.930744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.930666 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" event={"ID":"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd","Type":"ContainerStarted","Data":"d66717c9e586b1d57dacbe10f166abc08d67420f55dc93c3a6c3cf0a8b4bf1a2"} Nov 02 14:13:52.930744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.930718 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" event={"ID":"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd","Type":"ContainerStarted","Data":"041f02e31d4ae07ed580fad58be1dd4920cb2bb1c3fbb915e0533a1c26dd6743"} Nov 02 14:13:52.979574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:52.979522 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podStartSLOduration=1.9795087059999998 podStartE2EDuration="1.979508706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:52.977338104 +0000 UTC m=+1781.702959876" watchObservedRunningTime="2026-11-02 14:13:52.979508706 +0000 UTC m=+1781.705130476" Nov 02 14:13:53.325949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:53.325853 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:13:53.327180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:53.327156 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 14:13:53.327287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:53.327204 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 14:13:53.756006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:53.754029 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:54.326802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:54.326759 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 14:13:54.327314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:54.326818 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 14:13:54.904616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:54.904582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:55.187530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.187437 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw"] Nov 02 14:13:55.187714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.187656 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" podUID="b32c7ac4-b4a0-41a4-b14a-6de00fed3263" containerName="maas-api" containerID="cri-o://4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088" gracePeriod=30 Nov 02 14:13:55.319371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.319294 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:55.327350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.327322 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 14:13:55.327744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.327373 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 14:13:55.498688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.498517 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:55.741470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.740352 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:55.762115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.760893 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:13:55.929032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.928994 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-openshift-service-ca\") pod \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " Nov 02 14:13:55.929238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.929071 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-maas-api-tls\") pod \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " Nov 02 14:13:55.929333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.929261 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-kube-api-access-p2cqw\" (UniqueName: \"kubernetes.io/projected/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-kube-api-access-p2cqw\") pod \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " Nov 02 14:13:55.929393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.929335 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-metrics-certs\") pod \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\" (UID: \"b32c7ac4-b4a0-41a4-b14a-6de00fed3263\") " Nov 02 14:13:55.929449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.929385 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-openshift-service-ca" pod "b32c7ac4-b4a0-41a4-b14a-6de00fed3263" (UID: "b32c7ac4-b4a0-41a4-b14a-6de00fed3263"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:55.930716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.929790 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:55.956619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.956141 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-kube-api-access-p2cqw" pod "b32c7ac4-b4a0-41a4-b14a-6de00fed3263" (UID: "b32c7ac4-b4a0-41a4-b14a-6de00fed3263"). InnerVolumeSpecName "kube-api-access-p2cqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:55.957015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.956964 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-maas-api-tls" pod "b32c7ac4-b4a0-41a4-b14a-6de00fed3263" (UID: "b32c7ac4-b4a0-41a4-b14a-6de00fed3263"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:55.957558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.957534 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-metrics-certs" pod "b32c7ac4-b4a0-41a4-b14a-6de00fed3263" (UID: "b32c7ac4-b4a0-41a4-b14a-6de00fed3263"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:55.968802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.968771 5591 generic.go:356] "Generic (PLEG): container finished" podID="b32c7ac4-b4a0-41a4-b14a-6de00fed3263" containerID="4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088" exitCode=0 Nov 02 14:13:55.968917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.968865 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" event={"ID":"b32c7ac4-b4a0-41a4-b14a-6de00fed3263","Type":"ContainerDied","Data":"4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088"} Nov 02 14:13:55.968917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.968902 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" event={"ID":"b32c7ac4-b4a0-41a4-b14a-6de00fed3263","Type":"ContainerDied","Data":"b5bf263e51f4c0047e34525b503f9f7f53660e8f31f38d40333b96e79e715efe"} Nov 02 14:13:55.969012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.968922 5591 scope.go:122] "RemoveContainer" containerID="4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088" Nov 02 14:13:55.969078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.969065 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" Nov 02 14:13:55.981451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.981431 5591 scope.go:122] "RemoveContainer" containerID="4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088" Nov 02 14:13:55.981724 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:13:55.981701 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088\": container with ID starting with 4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088 not found: ID does not exist" containerID="4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088" Nov 02 14:13:55.981764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:55.981735 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088"} err="failed to get container status \"4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088\": rpc error: code = NotFound desc = could not find container \"4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088\": container with ID starting with 4313080e4becf853ff3950944ed9b88d3c2b26bfbd1b0a75bbb38993eda61088 not found: ID does not exist" Nov 02 14:13:56.071565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:56.034106 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2cqw\" (UniqueName: \"kubernetes.io/projected/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-kube-api-access-p2cqw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:56.071565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:56.034144 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:56.071565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:56.034158 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32c7ac4-b4a0-41a4-b14a-6de00fed3263-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:13:56.087851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:56.082438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:56.087851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:56.086033 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw"] Nov 02 14:13:56.087851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:56.086938 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw"] Nov 02 14:13:56.327046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:56.326533 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 14:13:56.327046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:56.326598 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 14:13:57.326348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:57.326312 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 14:13:57.326757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:57.326364 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 14:13:57.452286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:57.452239 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32c7ac4-b4a0-41a4-b14a-6de00fed3263" path="/var/lib/kubelet/pods/b32c7ac4-b4a0-41a4-b14a-6de00fed3263/volumes" Nov 02 14:13:58.179955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:58.179915 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:58.326999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:58.326961 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 14:13:58.327390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:58.327053 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 14:13:58.605185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:58.605100 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c07f7e-64954d94cb-bdpqw" podUID="b32c7ac4-b4a0-41a4-b14a-6de00fed3263" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.146:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:58.735565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:58.735524 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:59.084685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:59.084623 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:13:59.326645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:59.326605 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 14:13:59.326849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:13:59.326668 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 14:14:00.330823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.330791 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:14:00.331274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.331214 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:14:00.332055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.332035 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:14:00.543644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.543612 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:00.687763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.687729 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh"] Nov 02 14:14:00.688309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.688295 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b32c7ac4-b4a0-41a4-b14a-6de00fed3263" containerName="maas-api" Nov 02 14:14:00.688365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.688311 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32c7ac4-b4a0-41a4-b14a-6de00fed3263" containerName="maas-api" Nov 02 14:14:00.688409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.688400 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32c7ac4-b4a0-41a4-b14a-6de00fed3263" containerName="maas-api" Nov 02 14:14:00.691440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.691418 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:00.708031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.707973 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-2cddf2\"" Nov 02 14:14:00.708182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.708171 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-2cddf2-dockercfg-jjsfd\"" Nov 02 14:14:00.728751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.728720 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a5237-6a6a-4017-8107-f011ce4fe99e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh\" (UID: \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:00.729015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.728951 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7z9t\" (UniqueName: \"kubernetes.io/projected/ba4a5237-6a6a-4017-8107-f011ce4fe99e-kube-api-access-k7z9t\") pod \"payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh\" (UID: \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:00.735562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.735520 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh"] Nov 02 14:14:00.809949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.809912 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:00.857631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.840824 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a5237-6a6a-4017-8107-f011ce4fe99e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh\" (UID: \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:00.857631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.840910 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7z9t\" (UniqueName: \"kubernetes.io/projected/ba4a5237-6a6a-4017-8107-f011ce4fe99e-kube-api-access-k7z9t\") pod \"payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh\" (UID: \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:00.857631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.842784 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a5237-6a6a-4017-8107-f011ce4fe99e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh\" (UID: \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:00.857631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.854715 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7"] Nov 02 14:14:00.905261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.905134 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7z9t\" (UniqueName: \"kubernetes.io/projected/ba4a5237-6a6a-4017-8107-f011ce4fe99e-kube-api-access-k7z9t\") pod \"payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh\" (UID: \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:00.925123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.925088 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7"] Nov 02 14:14:00.925549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.925512 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:00.942697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.942619 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6sk\" (UniqueName: \"kubernetes.io/projected/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-kube-api-access-7t6sk\") pod \"payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7\" (UID: \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:00.942852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:00.942790 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7\" (UID: \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:01.003421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.003388 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:01.044473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.044430 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7\" (UID: \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:01.044707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.044664 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6sk\" (UniqueName: \"kubernetes.io/projected/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-kube-api-access-7t6sk\") pod \"payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7\" (UID: \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:01.045846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.045816 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7\" (UID: \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:01.093039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.092507 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6sk\" (UniqueName: \"kubernetes.io/projected/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-kube-api-access-7t6sk\") pod \"payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7\" (UID: \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:01.099292 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:14:01.099252 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:14:01.254397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.254309 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:01.311327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.311207 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh"] Nov 02 14:14:01.314218 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:14:01.314177 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4a5237_6a6a_4017_8107_f011ce4fe99e.slice/crio-ae31b1b43662fa5529fe4175811e5e1284874f2677dd0019bf7b842f006f4182 WatchSource:0}: Error finding container ae31b1b43662fa5529fe4175811e5e1284874f2677dd0019bf7b842f006f4182: Status 404 returned error can't find the container with id ae31b1b43662fa5529fe4175811e5e1284874f2677dd0019bf7b842f006f4182 Nov 02 14:14:01.579779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.579755 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7"] Nov 02 14:14:01.581574 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:14:01.581543 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae397ae7_a1b3_4f5b_be33_cbc33d08bb01.slice/crio-1b30812d1da2607a56a6dd118e97abdfc0355ae4c7b60ce9124f1daa963f214b WatchSource:0}: Error finding container 1b30812d1da2607a56a6dd118e97abdfc0355ae4c7b60ce9124f1daa963f214b: Status 404 returned error can't find the container with id 1b30812d1da2607a56a6dd118e97abdfc0355ae4c7b60ce9124f1daa963f214b Nov 02 14:14:01.776306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.776269 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:01.986907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:01.986870 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:02.033297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.033255 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" event={"ID":"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01","Type":"ContainerStarted","Data":"1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1"} Nov 02 14:14:02.033484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.033327 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:02.033484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.033348 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" event={"ID":"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01","Type":"ContainerStarted","Data":"1b30812d1da2607a56a6dd118e97abdfc0355ae4c7b60ce9124f1daa963f214b"} Nov 02 14:14:02.034927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.034898 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" event={"ID":"ba4a5237-6a6a-4017-8107-f011ce4fe99e","Type":"ContainerStarted","Data":"dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687"} Nov 02 14:14:02.035062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.034943 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" event={"ID":"ba4a5237-6a6a-4017-8107-f011ce4fe99e","Type":"ContainerStarted","Data":"ae31b1b43662fa5529fe4175811e5e1284874f2677dd0019bf7b842f006f4182"} Nov 02 14:14:02.035062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.034968 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:02.092742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.092690 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" podStartSLOduration=2.092674877 podStartE2EDuration="2.092674877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:02.091030977 +0000 UTC m=+1790.816652747" watchObservedRunningTime="2026-11-02 14:14:02.092674877 +0000 UTC m=+1790.818296624" Nov 02 14:14:02.138447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.138386 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" podStartSLOduration=2.13837446 podStartE2EDuration="2.13837446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:02.136858403 +0000 UTC m=+1790.862480173" watchObservedRunningTime="2026-11-02 14:14:02.13837446 +0000 UTC m=+1790.863996208" Nov 02 14:14:02.608170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.608130 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:02.805621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:02.805580 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:03.121242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.119448 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:03.334330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.334276 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:03.340203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.340174 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m"] Nov 02 14:14:03.343180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.343159 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.362265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.362239 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-4b7759-openshift-default-dockercfg-7nf54\"" Nov 02 14:14:03.380183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.379145 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m"] Nov 02 14:14:03.390391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390356 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-credential-socket\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.390556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390405 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gtrv\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-kube-api-access-8gtrv\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.390556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390439 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-socket\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.390683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390576 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-data\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.390683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390620 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-certs\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.390787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390689 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istiod-ca-cert\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.390844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390785 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-podinfo\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.390844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390832 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-envoy\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.390948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.390860 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-token\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492092 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-credential-socket\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492164 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gtrv\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-kube-api-access-8gtrv\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492239 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-socket\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492323 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-data\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492363 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-certs\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492432 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istiod-ca-cert\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492554 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-podinfo\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492598 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-envoy\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.492625 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-token\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.493491 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-data\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.493513 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-credential-socket\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.494156 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-certs\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.494801 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-socket\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.495009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.494811 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istiod-ca-cert\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.531331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.531294 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:03.558575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.558242 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-token\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.575008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.567821 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-podinfo\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.575008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.570326 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-envoy\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.585008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.577227 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gtrv\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-kube-api-access-8gtrv\") pod \"e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.693052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.692912 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:03.716838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:03.716796 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:04.052972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:04.052939 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m"] Nov 02 14:14:04.054296 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:14:04.054270 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3317507b_9f40_45b7_a3dc_9bfd1fc20c9c.slice/crio-92a4d77ff668bb4d638ebe907f13827a14a1f2b83ea4145fe6f7e5515c3e601a WatchSource:0}: Error finding container 92a4d77ff668bb4d638ebe907f13827a14a1f2b83ea4145fe6f7e5515c3e601a: Status 404 returned error can't find the container with id 92a4d77ff668bb4d638ebe907f13827a14a1f2b83ea4145fe6f7e5515c3e601a Nov 02 14:14:04.057382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:04.057349 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:04.057450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:04.057419 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:04.057489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:04.057453 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:04.079400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:04.079371 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" event={"ID":"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c","Type":"ContainerStarted","Data":"92a4d77ff668bb4d638ebe907f13827a14a1f2b83ea4145fe6f7e5515c3e601a"} Nov 02 14:14:04.427616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:04.427577 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:04.991374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:04.991343 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m"] Nov 02 14:14:04.994969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:04.994951 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.021020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.019544 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-960a68-openshift-default-dockercfg-b6hdg\"" Nov 02 14:14:05.028478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.028442 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:05.033818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.033791 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m"] Nov 02 14:14:05.034212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034187 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-envoy\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.034347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034326 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-istio-token\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.034477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034456 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253450c0-3bb5-4b4a-a312-63491e337372-istio-podinfo\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.034548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034534 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-credential-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.034606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034575 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253450c0-3bb5-4b4a-a312-63491e337372-istiod-ca-cert\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.034697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034683 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-certs\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.034768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034753 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.034833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034812 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-data\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.034885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.034847 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5hd\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-kube-api-access-5j5hd\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.143907 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253450c0-3bb5-4b4a-a312-63491e337372-istio-podinfo\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.143962 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-credential-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.144027 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253450c0-3bb5-4b4a-a312-63491e337372-istiod-ca-cert\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.144098 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-certs\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.144151 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.144187 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-data\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.144221 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5hd\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-kube-api-access-5j5hd\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.144254 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-envoy\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.144406 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-istio-token\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.146580 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.146944 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-credential-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.147756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.147693 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253450c0-3bb5-4b4a-a312-63491e337372-istiod-ca-cert\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.153092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.147955 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-certs\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.153092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.148648 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-data\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.192877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.192815 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" event={"ID":"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c","Type":"ContainerStarted","Data":"5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7"} Nov 02 14:14:05.214598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.214563 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253450c0-3bb5-4b4a-a312-63491e337372-istio-podinfo\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.215118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.215094 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-envoy\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.229238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.228176 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:05.231273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.231197 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-istio-token\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.248066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.245696 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5hd\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-kube-api-access-5j5hd\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.308105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.308064 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:05.634336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.634290 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" podStartSLOduration=2.634275205 podStartE2EDuration="2.634275205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:05.262320971 +0000 UTC m=+1793.987942742" watchObservedRunningTime="2026-11-02 14:14:05.634275205 +0000 UTC m=+1794.359896983" Nov 02 14:14:05.635559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.635536 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m"] Nov 02 14:14:05.636854 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:14:05.636831 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253450c0_3bb5_4b4a_a312_63491e337372.slice/crio-93df1760a46e0f370a8ff059fc5280286cbe3df4bd4f12c44fe699b08bac9d77 WatchSource:0}: Error finding container 93df1760a46e0f370a8ff059fc5280286cbe3df4bd4f12c44fe699b08bac9d77: Status 404 returned error can't find the container with id 93df1760a46e0f370a8ff059fc5280286cbe3df4bd4f12c44fe699b08bac9d77 Nov 02 14:14:05.639912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.639881 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:05.640010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.639943 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:05.640010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.639972 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:05.677560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.677532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:05.715559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.715528 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:05.721475 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.721445 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:05.800156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:05.799040 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:14:06.200650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.200616 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" event={"ID":"253450c0-3bb5-4b4a-a312-63491e337372","Type":"ContainerStarted","Data":"d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b"} Nov 02 14:14:06.201131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.200674 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" event={"ID":"253450c0-3bb5-4b4a-a312-63491e337372","Type":"ContainerStarted","Data":"93df1760a46e0f370a8ff059fc5280286cbe3df4bd4f12c44fe699b08bac9d77"} Nov 02 14:14:06.201131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.201103 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:06.202014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.201976 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:06.250963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.250915 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" podStartSLOduration=2.250901796 podStartE2EDuration="2.250901796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:06.248776121 +0000 UTC m=+1794.974397893" watchObservedRunningTime="2026-11-02 14:14:06.250901796 +0000 UTC m=+1794.976523565" Nov 02 14:14:06.308565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.308529 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:06.315219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.315188 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:06.444514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.444459 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:06.616060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:06.615676 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:07.061569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.061528 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w"] Nov 02 14:14:07.076621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.068302 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.165337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.165301 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w"] Nov 02 14:14:07.254359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.254332 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:07.256682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.256651 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.256901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.256880 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.257084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.257064 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5l7\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-kube-api-access-xv5l7\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.257225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.257209 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.257400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.257383 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.257530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.257513 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.257711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.257694 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.257836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.257820 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.258077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.258043 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.261273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.261253 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:07.491063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.490818 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.491063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.490890 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.523070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.522343 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.527011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.526345 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.527011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.526411 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.527011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.526453 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.527011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.526559 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.527011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.526779 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.527011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.526863 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.527011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.526902 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5l7\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-kube-api-access-xv5l7\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.527011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.526947 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.531012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.527925 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.535306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.535277 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.572021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.537650 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.655368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.654666 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.655614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.654820 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.727696 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.726521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:07.760401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.760303 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5l7\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-kube-api-access-xv5l7\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:07.970669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:07.946019 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:08.044485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.042037 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:08.228007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.227253 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:08.547030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.544103 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:08.547030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.544221 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:08.574020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.573800 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:14:08.610549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.575177 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:08.610549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.575469 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:14:08.610549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.575812 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:14:08.679005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.672712 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:08.699150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.699097 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:08.738772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.738717 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:09.007022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:08.998729 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:09.211356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.211152 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:09.559491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.558439 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:09.585067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.580125 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:09.594128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.594078 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:14:09.723008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.716659 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:14:09.723008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.716730 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:14:09.723008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.716766 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w"] Nov 02 14:14:09.723008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.716824 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:14:09.781635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.750671 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:09.781635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.751396 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:14:09.781635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.766283 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:09.781635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.766383 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:09.781635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.766431 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:09.903551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.903521 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:14:09.950431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:09.950397 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:14:10.016404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:10.015824 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:14:10.016404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:10.015902 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:14:10.336005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:10.335904 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" event={"ID":"47c22a06-aa23-4cf5-a58f-c75ae82225b5","Type":"ContainerStarted","Data":"7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8"} Nov 02 14:14:10.336005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:10.335968 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" event={"ID":"47c22a06-aa23-4cf5-a58f-c75ae82225b5","Type":"ContainerStarted","Data":"1f0cb08a6aa11752de751e21372fcd19d4d9a0a66b3f369a1bd1bfe05c4b396d"} Nov 02 14:14:10.385964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:10.385904 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podStartSLOduration=3.385887236 podStartE2EDuration="3.385887236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:10.38247328 +0000 UTC m=+1799.108095049" watchObservedRunningTime="2026-11-02 14:14:10.385887236 +0000 UTC m=+1799.111509009" Nov 02 14:14:10.857864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:10.857817 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:11.088706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:11.088620 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:11.264975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:11.264941 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:11.348588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:11.348312 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:14:11.348588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:11.348371 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:14:11.870680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:11.870645 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:14:11.894091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:11.894061 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:14:12.228150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:12.228112 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:14:12.228313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:12.228194 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:14:12.469701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:12.469668 5591 scope.go:122] "RemoveContainer" containerID="53192e5c0494562ecc83dbd3537a96c04d851af2570051399365c6bac1df0f4f" Nov 02 14:14:12.726278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:12.726241 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:12.907412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:12.907370 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:13.131587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:13.128597 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:13.160396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:13.160304 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:14:13.216371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:13.204406 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:14:13.296008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:13.254332 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:14:13.319776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:13.313779 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:14:14.228658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:14.228624 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:14:14.229103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:14.228679 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:14:14.859421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:14.859374 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:15.073280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:15.073247 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:15.230901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:15.230865 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:14:15.262244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:15.262206 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:14:15.262836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:15.261836 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:16.228210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.228173 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:14:16.228395 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.228224 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:14:16.647470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.647419 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:16.729320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.729280 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh"] Nov 02 14:14:16.735391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.735365 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.760014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.759954 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh"] Nov 02 14:14:16.803699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.803657 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.803875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.803791 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.803875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.803848 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.804083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.803888 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.804083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.804037 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jw47\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-kube-api-access-4jw47\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.804202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.804110 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.804262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.804233 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.804317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.804293 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.804386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.804366 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906017 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906102 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906161 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906198 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906260 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906301 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906341 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906454 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jw47\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-kube-api-access-4jw47\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906560 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906576 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906602 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906801 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.906846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.906810 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.907643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.907613 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.908602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.908580 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.908961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.908942 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.930845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.930813 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:16.931081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:16.931054 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jw47\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-kube-api-access-4jw47\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:17.059007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:17.056314 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:17.228345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:17.228307 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:14:17.228523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:17.228377 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:14:17.473573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:17.473545 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh"] Nov 02 14:14:17.475753 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:14:17.475717 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfd2773_c8bc_4ec8_8a02_210b4ff2bb05.slice/crio-e08b8a3d90645693248ec7ecfec2a2b2e5c2f3e1affbd576fbcae2242b8b134e WatchSource:0}: Error finding container e08b8a3d90645693248ec7ecfec2a2b2e5c2f3e1affbd576fbcae2242b8b134e: Status 404 returned error can't find the container with id e08b8a3d90645693248ec7ecfec2a2b2e5c2f3e1affbd576fbcae2242b8b134e Nov 02 14:14:17.478809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:17.478774 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:17.478904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:17.478838 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:17.478904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:17.478868 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:18.245012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:18.238321 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:18.245012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:18.239833 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:18.252318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:18.252283 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:14:18.423563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:18.423531 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" event={"ID":"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05","Type":"ContainerStarted","Data":"f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971"} Nov 02 14:14:18.423764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:18.423582 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" event={"ID":"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05","Type":"ContainerStarted","Data":"e08b8a3d90645693248ec7ecfec2a2b2e5c2f3e1affbd576fbcae2242b8b134e"} Nov 02 14:14:18.466821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:18.466784 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:18.509680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:18.508080 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" podStartSLOduration=2.508056064 podStartE2EDuration="2.508056064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:18.474594582 +0000 UTC m=+1807.200216365" watchObservedRunningTime="2026-11-02 14:14:18.508056064 +0000 UTC m=+1807.233677837" Nov 02 14:14:19.057280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:19.057235 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:19.058842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:19.058813 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 14:14:19.058961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:19.058861 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 14:14:19.258659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:19.258618 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:20.057516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:20.057468 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 14:14:20.057696 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:20.057540 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 14:14:20.343121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:20.342949 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:21.057563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:21.057519 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 14:14:21.057790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:21.057586 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 14:14:22.057559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.057517 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 14:14:22.058236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.058198 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 14:14:22.115229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.115187 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk"] Nov 02 14:14:22.119839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.119805 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:22.126034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.125413 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.185460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.185426 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk"] Nov 02 14:14:22.213036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.212998 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.213199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.213054 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.213243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.213219 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.213276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.213263 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.213345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.213286 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.213423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.213408 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.213474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.213461 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwc9l\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-kube-api-access-nwc9l\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.213608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.213588 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.213680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.213635 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.314706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.314605 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.314881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.314724 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.314881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.314782 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.314881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.314806 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.314881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.314873 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.315128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.314940 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwc9l\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-kube-api-access-nwc9l\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.315128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.315061 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.315128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.315106 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.315277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.315149 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.315340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.315289 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.315416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.315380 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.315588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.315559 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.315665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.315649 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.316050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.316026 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.317174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.317156 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.317744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.317720 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.358637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.358569 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.358802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.358764 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwc9l\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-kube-api-access-nwc9l\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.502265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.502233 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:22.857914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.857884 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk"] Nov 02 14:14:22.860394 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:14:22.860357 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0349d3_72dc_4a9e_9fb6_30a622dbb3c3.slice/crio-f1d91e8d7a468692305bea9a111edeb9d0802da8cd81d61b92c3d37ea71aa8dc WatchSource:0}: Error finding container f1d91e8d7a468692305bea9a111edeb9d0802da8cd81d61b92c3d37ea71aa8dc: Status 404 returned error can't find the container with id f1d91e8d7a468692305bea9a111edeb9d0802da8cd81d61b92c3d37ea71aa8dc Nov 02 14:14:22.863569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.863173 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:22.863569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.863238 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:22.863569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:22.863266 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:23.059502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.059465 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 14:14:23.060179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.060129 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 14:14:23.412328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.412291 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:23.480216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.480175 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" event={"ID":"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3","Type":"ContainerStarted","Data":"a241e61dc0a74015f71286938df5992201485b6a7ceb36318e1428a6c5d8fe02"} Nov 02 14:14:23.480444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.480236 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" event={"ID":"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3","Type":"ContainerStarted","Data":"f1d91e8d7a468692305bea9a111edeb9d0802da8cd81d61b92c3d37ea71aa8dc"} Nov 02 14:14:23.503020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.502975 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:23.505049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.505020 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:23.505189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.505074 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:23.538547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.538490 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podStartSLOduration=1.5384729369999999 podStartE2EDuration="1.538472937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:23.530457774 +0000 UTC m=+1812.256079545" watchObservedRunningTime="2026-11-02 14:14:23.538472937 +0000 UTC m=+1812.264094706" Nov 02 14:14:23.716039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.715860 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2"] Nov 02 14:14:23.722186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.722153 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.750381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.750338 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2"] Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857037 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857112 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857223 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857268 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857319 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857357 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857381 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvsqf\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-kube-api-access-dvsqf\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857472 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.857539 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.862077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.858414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:23.961896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.961841 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.963566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.963541 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.970253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.970162 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.971098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.971068 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.980209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.980155 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.980922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.980897 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:23.990009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.981468 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.011015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.007105 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvsqf\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-kube-api-access-dvsqf\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.011015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.007266 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.011015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.981396 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.011015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.983488 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.011015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.963392 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.011015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.980788 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.011015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:23.963975 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.011015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.008561 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.016185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.013086 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.047027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.046968 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:24.064008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.057442 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvsqf\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-kube-api-access-dvsqf\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.064773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.064740 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 14:14:24.065006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.064960 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 14:14:24.071665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.071625 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.350373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.350265 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:24.503394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.503349 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:24.503598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.503406 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:24.753018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.750615 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2"] Nov 02 14:14:24.795793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.795734 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:24.796177 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.796148 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:24.796376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:24.796344 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:25.463008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.438461 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:25.512434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.506168 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:25.512434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.506249 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:25.646010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.635210 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:25.688061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.687113 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" event={"ID":"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe","Type":"ContainerStarted","Data":"4bed9bed4bfb58e996ec53adb61c620e55b227c1fbef32f004099b5bcb78d230"} Nov 02 14:14:25.688061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.687175 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" event={"ID":"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe","Type":"ContainerStarted","Data":"3388d176b785b3943bd3dbe53b858c5bf3255b4979f29f67d81af8c746a22fac"} Nov 02 14:14:25.780199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.780075 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podStartSLOduration=2.780060212 podStartE2EDuration="2.780060212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:25.772448032 +0000 UTC m=+1814.498069804" watchObservedRunningTime="2026-11-02 14:14:25.780060212 +0000 UTC m=+1814.505681982" Nov 02 14:14:25.793179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.793144 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:25.793768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.793742 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:25.796067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.796042 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:14:25.994006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.987436 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c"] Nov 02 14:14:25.994006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:25.992118 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.023012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.014548 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-960a68-serving-cert\"" Nov 02 14:14:26.023012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.018551 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c"] Nov 02 14:14:26.369106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.319824 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:26.454721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.403953 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:26.454721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.452536 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:14:26.454721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.452597 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:14:26.501007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.475359 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.512028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.508265 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2w2\" (UniqueName: \"kubernetes.io/projected/afa63769-17d5-4169-96f0-f586c739a12e-kube-api-access-zj2w2\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.512028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.508334 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.512028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.508368 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa63769-17d5-4169-96f0-f586c739a12e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.512028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.507193 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:26.512028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.508736 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:26.523102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.523067 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:26.610315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.610269 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.610493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.610389 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2w2\" (UniqueName: \"kubernetes.io/projected/afa63769-17d5-4169-96f0-f586c739a12e-kube-api-access-zj2w2\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.610493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.610418 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.610493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.610434 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa63769-17d5-4169-96f0-f586c739a12e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.611099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.611065 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa63769-17d5-4169-96f0-f586c739a12e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.613426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.613406 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.618660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.618633 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.659067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.659030 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2w2\" (UniqueName: \"kubernetes.io/projected/afa63769-17d5-4169-96f0-f586c739a12e-kube-api-access-zj2w2\") pod \"maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:26.908355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:26.908130 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:27.298078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:27.294836 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c"] Nov 02 14:14:27.363867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:27.363396 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:14:27.363867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:27.363469 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:14:27.496008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:27.486957 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:27.579206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:27.569253 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:27.579206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:27.569317 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:27.704165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:27.704126 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" event={"ID":"afa63769-17d5-4169-96f0-f586c739a12e","Type":"ContainerStarted","Data":"d66a72212cc1cd6ffc933105c9ef783f0175d61a25ace129e9f580f0ba03c48a"} Nov 02 14:14:28.185011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.176634 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:28.365400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.365362 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:14:28.365660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.365632 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:14:28.456062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.455795 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:28.519082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.517278 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:28.519082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.517355 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:28.659667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.659558 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:28.712333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.712229 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" event={"ID":"afa63769-17d5-4169-96f0-f586c739a12e","Type":"ContainerStarted","Data":"56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e"} Nov 02 14:14:28.712333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.712303 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:28.785772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.784075 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" podStartSLOduration=3.377210865 podStartE2EDuration="3.784054277s" totalImagesPullingTime="406.843412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:28.762243044 +0000 UTC m=+1817.487864815" watchObservedRunningTime="2026-11-02 14:14:28.784054277 +0000 UTC m=+1817.509676048" Nov 02 14:14:28.928082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:28.928042 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:29.356510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:29.356470 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:14:29.356833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:29.356789 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:14:29.520614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:29.519798 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:29.520614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:29.519864 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:30.368003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:30.367782 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:14:30.368003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:30.367846 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:14:30.511018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:30.510124 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:30.511018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:30.510192 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:31.201264 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:14:31.201219 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:14:31.357612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:31.357575 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:31.375504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:31.375456 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:14:31.376117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:31.376081 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:14:31.504773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:31.504674 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 14:14:31.505087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:31.505031 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 14:14:32.233717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:32.233674 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:32.354891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:32.354404 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:14:32.354891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:32.354482 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:14:32.399012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:32.396058 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:14:32.447090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:32.443138 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:32.612604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:32.612513 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:32.613049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:32.613019 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:32.614278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:32.614250 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:14:33.417474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:33.417439 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:33.418466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:33.418443 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:33.419485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:33.419448 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:33.762133 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:33.761026 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:34.000473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:34.000437 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:34.264153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:34.264113 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:34.773677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:34.773651 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:14:35.550289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:35.550249 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:35.719887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:35.719851 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:37.288018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:37.287949 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:37.723339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:37.723301 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:37.951751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:37.951707 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:39.241269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:39.241219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:39.404977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:39.404936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:39.731753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:39.731720 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:40.256878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:40.256845 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:40.675556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:40.675519 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:40.798250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:40.798213 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk"] Nov 02 14:14:40.798511 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:40.798474 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" podUID="7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" containerName="maas-api" containerID="cri-o://11a25eaa933d2fb417e0a8b97617f5ed77e623065ff25d39b1053acb455f2a66" gracePeriod=30 Nov 02 14:14:40.922148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:40.922120 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:40.952383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:40.952294 5591 generic.go:356] "Generic (PLEG): container finished" podID="7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" containerID="11a25eaa933d2fb417e0a8b97617f5ed77e623065ff25d39b1053acb455f2a66" exitCode=0 Nov 02 14:14:40.952531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:40.952379 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" event={"ID":"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e","Type":"ContainerDied","Data":"11a25eaa933d2fb417e0a8b97617f5ed77e623065ff25d39b1053acb455f2a66"} Nov 02 14:14:41.131746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.131411 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:14:41.246070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.245754 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-metrics-certs\") pod \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " Nov 02 14:14:41.246070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.245878 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-kube-api-access-6gzb4\" (UniqueName: \"kubernetes.io/projected/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-kube-api-access-6gzb4\") pod \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " Nov 02 14:14:41.246070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.245914 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-openshift-service-ca\") pod \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " Nov 02 14:14:41.246070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.246020 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls\") pod \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\" (UID: \"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e\") " Nov 02 14:14:41.248435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.247685 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-openshift-service-ca" pod "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" (UID: "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:41.248575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.248469 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:41.252353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.252301 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-metrics-certs" pod "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" (UID: "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:14:41.253332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.253300 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-kube-api-access-6gzb4" pod "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" (UID: "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e"). InnerVolumeSpecName "kube-api-access-6gzb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:41.258780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.258743 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls" pod "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" (UID: "7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:14:41.363018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.360808 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:41.363018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.360847 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gzb4\" (UniqueName: \"kubernetes.io/projected/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-kube-api-access-6gzb4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:41.363018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.360863 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:41.739855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.737482 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:41.970401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.970361 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" event={"ID":"7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e","Type":"ContainerDied","Data":"287c60a4fcfe9e65f9443e396dcee518272ac60dfa8d6d5dc66cdc912f2385da"} Nov 02 14:14:41.970606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.970368 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk" Nov 02 14:14:41.970606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:41.970459 5591 scope.go:122] "RemoveContainer" containerID="11a25eaa933d2fb417e0a8b97617f5ed77e623065ff25d39b1053acb455f2a66" Nov 02 14:14:42.027710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:42.027680 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk"] Nov 02 14:14:42.039178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:42.039152 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cdaf64-5867495947-qp9hk"] Nov 02 14:14:42.487011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:42.483804 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:42.687371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:42.687332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:43.208260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.208182 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:43.432761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.432718 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" path="/var/lib/kubelet/pods/7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e/volumes" Nov 02 14:14:43.502193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.502118 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99"] Nov 02 14:14:43.502601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.502320 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" containerID="cri-o://3c46a1091706577a99546d331b68cadd65d823d8313504c7dd78a087d764852e" gracePeriod=30 Nov 02 14:14:43.543843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.543812 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2"] Nov 02 14:14:43.544049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.544028 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" containerID="cri-o://4bed9bed4bfb58e996ec53adb61c620e55b227c1fbef32f004099b5bcb78d230" gracePeriod=30 Nov 02 14:14:43.611953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.611921 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:43.810065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.809381 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:14:43.827274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:43.827232 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:44.598852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:44.598800 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:44.819222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:44.819170 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:46.237194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:46.237147 5591 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:14:46.237833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:46.237795 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:46.442911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:46.442870 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:46.676009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:46.674269 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:46.949271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:46.949160 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:47.159009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:47.157135 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:47.896600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:47.896561 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:48.084114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.084067 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm"] Nov 02 14:14:48.085439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.085411 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" containerName="maas-api" Nov 02 14:14:48.085637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.085621 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" containerName="maas-api" Nov 02 14:14:48.086456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.086429 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea3a05-b1cf-4c9f-8c37-9b0fff5eef7e" containerName="maas-api" Nov 02 14:14:48.088579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.088531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:48.092432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.092398 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm"] Nov 02 14:14:48.092550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.092515 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:48.123161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.123130 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-4c2811-dockercfg-j6czh\"" Nov 02 14:14:48.160094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.157407 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-4c2811\"" Nov 02 14:14:48.329639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.329603 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nchd2\" (UniqueName: \"kubernetes.io/projected/e254d3cf-7490-4cf8-b709-493a2ee68794-kube-api-access-nchd2\") pod \"payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm\" (UID: \"e254d3cf-7490-4cf8-b709-493a2ee68794\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:48.374263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.330644 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl"] Nov 02 14:14:48.446571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.444901 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:48.467606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.465306 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:14:48.467606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.465388 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:14:48.522104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.513516 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e254d3cf-7490-4cf8-b709-493a2ee68794-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm\" (UID: \"e254d3cf-7490-4cf8-b709-493a2ee68794\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:48.527111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.522506 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl"] Nov 02 14:14:48.527111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.523319 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:48.618066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.615333 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxccb\" (UniqueName: \"kubernetes.io/projected/c10e334b-2678-4fe5-a5da-16d9bd9890eb-kube-api-access-wxccb\") pod \"payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl\" (UID: \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:48.618066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.615484 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e334b-2678-4fe5-a5da-16d9bd9890eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl\" (UID: \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:48.618066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.615568 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nchd2\" (UniqueName: \"kubernetes.io/projected/e254d3cf-7490-4cf8-b709-493a2ee68794-kube-api-access-nchd2\") pod \"payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm\" (UID: \"e254d3cf-7490-4cf8-b709-493a2ee68794\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:48.618066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.615652 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e254d3cf-7490-4cf8-b709-493a2ee68794-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm\" (UID: \"e254d3cf-7490-4cf8-b709-493a2ee68794\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:48.618388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.618096 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e254d3cf-7490-4cf8-b709-493a2ee68794-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm\" (UID: \"e254d3cf-7490-4cf8-b709-493a2ee68794\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:48.674753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.667511 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:48.676724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.676656 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nchd2\" (UniqueName: \"kubernetes.io/projected/e254d3cf-7490-4cf8-b709-493a2ee68794-kube-api-access-nchd2\") pod \"payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm\" (UID: \"e254d3cf-7490-4cf8-b709-493a2ee68794\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:48.696061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.696013 5591 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 14:14:48.696231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.696081 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 14:14:48.718261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.718215 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxccb\" (UniqueName: \"kubernetes.io/projected/c10e334b-2678-4fe5-a5da-16d9bd9890eb-kube-api-access-wxccb\") pod \"payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl\" (UID: \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:48.718739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.718714 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e334b-2678-4fe5-a5da-16d9bd9890eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl\" (UID: \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:48.719632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.719608 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e334b-2678-4fe5-a5da-16d9bd9890eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl\" (UID: \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:48.793023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.784747 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxccb\" (UniqueName: \"kubernetes.io/projected/c10e334b-2678-4fe5-a5da-16d9bd9890eb-kube-api-access-wxccb\") pod \"payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl\" (UID: \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:48.846341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:48.846300 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:49.002279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.001974 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:49.003084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.002917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:49.496840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.496805 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:49.608425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.608368 5591 generic.go:356] "Generic (PLEG): container finished" podID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerID="4bed9bed4bfb58e996ec53adb61c620e55b227c1fbef32f004099b5bcb78d230" exitCode=0 Nov 02 14:14:49.609262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.608855 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" event={"ID":"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe","Type":"ContainerDied","Data":"4bed9bed4bfb58e996ec53adb61c620e55b227c1fbef32f004099b5bcb78d230"} Nov 02 14:14:49.614369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.614330 5591 generic.go:356] "Generic (PLEG): container finished" podID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerID="3c46a1091706577a99546d331b68cadd65d823d8313504c7dd78a087d764852e" exitCode=0 Nov 02 14:14:49.614620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.614539 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" event={"ID":"8eb599d3-ebb0-419d-806d-d4a34ad60032","Type":"ContainerDied","Data":"3c46a1091706577a99546d331b68cadd65d823d8313504c7dd78a087d764852e"} Nov 02 14:14:49.614791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.614776 5591 scope.go:122] "RemoveContainer" containerID="eea33cfe63e8d74660a27e236eb08ccb338fffddfbc95b9f2739ffc220551771" Nov 02 14:14:49.714246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.713477 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:49.816089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.816056 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:14:49.818595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.818564 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-certs\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.818754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.818638 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-kube-api-access-sqsz7\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-kube-api-access-sqsz7\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.818754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.818686 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-envoy\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.818754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.818714 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-credential-socket\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.818935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.818766 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-podinfo\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.818935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.818929 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-data\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.819085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.819008 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb599d3-ebb0-419d-806d-d4a34ad60032-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb599d3-ebb0-419d-806d-d4a34ad60032-istiod-ca-cert\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.819085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.819075 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-socket\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.819206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.819103 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-token\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-token\") pod \"8eb599d3-ebb0-419d-806d-d4a34ad60032\" (UID: \"8eb599d3-ebb0-419d-806d-d4a34ad60032\") " Nov 02 14:14:49.819444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.819413 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-credential-socket" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.819552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.819450 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-data" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.819723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.819699 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.819796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.819731 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.819966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.819942 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-certs" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.821237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.820403 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-socket" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.821237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.820458 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb599d3-ebb0-419d-806d-d4a34ad60032-istiod-ca-cert" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:49.823212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.823166 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-kube-api-access-sqsz7" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "kube-api-access-sqsz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:49.823649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.823623 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-envoy" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.825697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.825619 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:49.830198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.830170 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-token" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:49.881023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.860160 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-podinfo" pod "8eb599d3-ebb0-419d-806d-d4a34ad60032" (UID: "8eb599d3-ebb0-419d-806d-d4a34ad60032"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:14:49.922873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.922018 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-token\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-token\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.922873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.922117 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-envoy\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.922873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.922148 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-certs\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.923163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923124 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-podinfo\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.924025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923262 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-kube-api-access-dvsqf\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-kube-api-access-dvsqf\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.924025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923299 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-socket\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.924025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923335 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-credential-socket\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.924025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923384 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istiod-ca-cert\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.924025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923438 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-data\") pod \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\" (UID: \"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe\") " Nov 02 14:14:49.924025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923743 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-socket" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.924025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923788 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-certs" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.924025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.923965 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-credential-socket" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924194 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924216 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924231 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eb599d3-ebb0-419d-806d-d4a34ad60032-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924246 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924260 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924273 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924287 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqsz7\" (UniqueName: \"kubernetes.io/projected/8eb599d3-ebb0-419d-806d-d4a34ad60032-kube-api-access-sqsz7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924300 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924312 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924324 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eb599d3-ebb0-419d-806d-d4a34ad60032-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:49.924610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.924268 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istiod-ca-cert" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:49.925378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.925341 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-data" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.929359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.929333 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-token" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:49.933213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.929812 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-envoy" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:14:49.933213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.929941 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-kube-api-access-dvsqf" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "kube-api-access-dvsqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:50.001009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:49.986141 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-podinfo" pod "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" (UID: "82dbfeb9-14ed-4e2d-9071-a17e8ca021fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:14:50.033012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.029793 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl"] Nov 02 14:14:50.055073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.052137 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:50.055073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.052184 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:50.055073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.052198 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:50.055073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.052213 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvsqf\" (UniqueName: \"kubernetes.io/projected/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-kube-api-access-dvsqf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:50.055073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.052228 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:50.055073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.052240 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:14:50.125349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.125305 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm"] Nov 02 14:14:50.299252 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:14:50.298710 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode254d3cf_7490_4cf8_b709_493a2ee68794.slice/crio-0c5e33090fc8bdc89c4ebf3832225e512a4a8747225144411e8e8afebe7103d3 WatchSource:0}: Error finding container 0c5e33090fc8bdc89c4ebf3832225e512a4a8747225144411e8e8afebe7103d3: Status 404 returned error can't find the container with id 0c5e33090fc8bdc89c4ebf3832225e512a4a8747225144411e8e8afebe7103d3 Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.388371 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c"] Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389350 5591 cpu_manager.go:414] "containerMap: removing container" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389373 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389395 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389404 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389429 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389437 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389600 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389623 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" containerName="istio-proxy" Nov 02 14:14:50.392346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.389634 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" containerName="istio-proxy" Nov 02 14:14:50.405361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.396449 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.405361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.400891 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c"] Nov 02 14:14:50.423868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.423834 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fcefcc-openshift-default-dockercfg-r7kkx\"" Nov 02 14:14:50.429868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.429838 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:50.559094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.558628 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.559094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.558724 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.559094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.558804 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.559094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.558865 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.559094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.558973 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndnf\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-kube-api-access-6ndnf\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.571020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.560064 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.571020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.560169 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.571020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.560473 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.571020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.560529 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.692009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.682044 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.710688 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.710810 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.710924 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.710959 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.711010 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.711040 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.711198 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndnf\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-kube-api-access-6ndnf\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.711231 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.711270 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.712254 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.712544 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.715769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.715744 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.716233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.716197 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.716378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.716359 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.718505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.718461 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.720178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.719423 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" Nov 02 14:14:50.720178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.719668 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2" event={"ID":"82dbfeb9-14ed-4e2d-9071-a17e8ca021fe","Type":"ContainerDied","Data":"3388d176b785b3943bd3dbe53b858c5bf3255b4979f29f67d81af8c746a22fac"} Nov 02 14:14:50.720178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.719726 5591 scope.go:122] "RemoveContainer" containerID="4bed9bed4bfb58e996ec53adb61c620e55b227c1fbef32f004099b5bcb78d230" Nov 02 14:14:50.721557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.720615 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.724389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.723867 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" event={"ID":"c10e334b-2678-4fe5-a5da-16d9bd9890eb","Type":"ContainerStarted","Data":"1e3469dc0466101a4f506e19ccb34fc787e776710c03ecc315c2bdaac87a6e0c"} Nov 02 14:14:50.727887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.726363 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" Nov 02 14:14:50.727887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.726387 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99" event={"ID":"8eb599d3-ebb0-419d-806d-d4a34ad60032","Type":"ContainerDied","Data":"341b253c11a51a183d7565e2c70b0f9b03cb4928333c73985be8f244b8d7e348"} Nov 02 14:14:50.731148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.731123 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" event={"ID":"e254d3cf-7490-4cf8-b709-493a2ee68794","Type":"ContainerStarted","Data":"0c5e33090fc8bdc89c4ebf3832225e512a4a8747225144411e8e8afebe7103d3"} Nov 02 14:14:50.792669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.792592 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.795971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.795939 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndnf\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-kube-api-access-6ndnf\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.800343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.800282 5591 scope.go:122] "RemoveContainer" containerID="3c46a1091706577a99546d331b68cadd65d823d8313504c7dd78a087d764852e" Nov 02 14:14:50.901004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.897065 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:50.913009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.906381 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:50.927270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.912577 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99"] Nov 02 14:14:50.927270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.915709 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-hcw99"] Nov 02 14:14:50.996248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.996204 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2"] Nov 02 14:14:50.999264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:50.999233 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-9kwb2"] Nov 02 14:14:51.392427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.392213 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c"] Nov 02 14:14:51.398030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.397999 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:51.398189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.398174 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:51.398279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.398267 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:51.477008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.475970 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dbfeb9-14ed-4e2d-9071-a17e8ca021fe" path="/var/lib/kubelet/pods/82dbfeb9-14ed-4e2d-9071-a17e8ca021fe/volumes" Nov 02 14:14:51.496005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.484531 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb599d3-ebb0-419d-806d-d4a34ad60032" path="/var/lib/kubelet/pods/8eb599d3-ebb0-419d-806d-d4a34ad60032/volumes" Nov 02 14:14:51.798074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.798040 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" event={"ID":"c10e334b-2678-4fe5-a5da-16d9bd9890eb","Type":"ContainerStarted","Data":"3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66"} Nov 02 14:14:51.799527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.799492 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:14:51.804495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.803965 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" event={"ID":"e254d3cf-7490-4cf8-b709-493a2ee68794","Type":"ContainerStarted","Data":"fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932"} Nov 02 14:14:51.804495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.804208 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:14:51.806599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.806571 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" event={"ID":"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7","Type":"ContainerStarted","Data":"de630999105654b44dd9a3973428dc8f23ca70d8fbd7ab7336ffcc8129536ee0"} Nov 02 14:14:51.806741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.806608 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" event={"ID":"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7","Type":"ContainerStarted","Data":"7ac5ccbb8d6bc875697905af1daa8b3c67fa473ac4b8ef3a7dbb4441bea0329e"} Nov 02 14:14:51.850010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.842139 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" podStartSLOduration=3.842122101 podStartE2EDuration="3.842122101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:51.841364286 +0000 UTC m=+1840.566986057" watchObservedRunningTime="2026-11-02 14:14:51.842122101 +0000 UTC m=+1840.567743872" Nov 02 14:14:51.905901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.904857 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" podStartSLOduration=3.904844146 podStartE2EDuration="3.904844146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:51.904230695 +0000 UTC m=+1840.629852480" watchObservedRunningTime="2026-11-02 14:14:51.904844146 +0000 UTC m=+1840.630465915" Nov 02 14:14:51.910545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.910520 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:51.954225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:51.954159 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podStartSLOduration=1.954140848 podStartE2EDuration="1.954140848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:51.953623371 +0000 UTC m=+1840.679245141" watchObservedRunningTime="2026-11-02 14:14:51.954140848 +0000 UTC m=+1840.679762619" Nov 02 14:14:52.265419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:52.265283 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:52.916858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:52.916826 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:53.823598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:53.823565 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:53.824805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:53.824777 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:14:54.126402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:54.122687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:54.331337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:54.331267 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:54.530819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:54.530782 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:55.974996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:55.974823 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:56.144211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.144174 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:56.855806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.855708 5591 generic.go:356] "Generic (PLEG): container finished" podID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerID="5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7" exitCode=0 Nov 02 14:14:56.855806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.855783 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" event={"ID":"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c","Type":"ContainerDied","Data":"5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7"} Nov 02 14:14:56.856316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.856292 5591 scope.go:122] "RemoveContainer" containerID="5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7" Nov 02 14:14:56.857524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.857504 5591 generic.go:356] "Generic (PLEG): container finished" podID="253450c0-3bb5-4b4a-a312-63491e337372" containerID="d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b" exitCode=0 Nov 02 14:14:56.857630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.857580 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" event={"ID":"253450c0-3bb5-4b4a-a312-63491e337372","Type":"ContainerDied","Data":"d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b"} Nov 02 14:14:56.857948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.857934 5591 scope.go:122] "RemoveContainer" containerID="d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b" Nov 02 14:14:56.858035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.857942 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:56.858035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.858023 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:56.858116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.858056 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:56.859629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.859599 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:56.859698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.859681 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:56.859746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:56.859730 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:14:57.859006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:57.858624 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:57.892340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:57.892028 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" event={"ID":"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c","Type":"ContainerStarted","Data":"f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4"} Nov 02 14:14:57.894602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:57.894571 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" event={"ID":"253450c0-3bb5-4b4a-a312-63491e337372","Type":"ContainerStarted","Data":"5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4"} Nov 02 14:14:58.309279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:58.309245 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:14:58.309549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:58.309529 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:14:58.309671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:58.309653 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:14:58.714832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:58.714797 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:14:58.715064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:58.714931 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 02 14:14:58.715064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:58.714968 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 02 14:14:59.309067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:59.309029 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:14:59.309488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:59.309082 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:14:59.591132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:59.590822 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:14:59.715037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:59.714890 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 02 14:14:59.715037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:14:59.714955 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 02 14:15:00.178636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.178595 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2"] Nov 02 14:15:00.186076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.186044 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:00.206356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.206330 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m"] Nov 02 14:15:00.209329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.209292 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w484\" (UniqueName: \"kubernetes.io/projected/3ce72d82-b7ce-44e5-8767-4249023d620c-kube-api-access-7w484\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2\" (UID: \"3ce72d82-b7ce-44e5-8767-4249023d620c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:00.209543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.209522 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce72d82-b7ce-44e5-8767-4249023d620c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2\" (UID: \"3ce72d82-b7ce-44e5-8767-4249023d620c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:00.211014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.210976 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:00.227672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.227639 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2"] Nov 02 14:15:00.294933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.291338 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\"" Nov 02 14:15:00.295155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.294355 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m"] Nov 02 14:15:00.310922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.310868 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce72d82-b7ce-44e5-8767-4249023d620c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2\" (UID: \"3ce72d82-b7ce-44e5-8767-4249023d620c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:00.311457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.311101 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w484\" (UniqueName: \"kubernetes.io/projected/3ce72d82-b7ce-44e5-8767-4249023d620c-kube-api-access-7w484\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2\" (UID: \"3ce72d82-b7ce-44e5-8767-4249023d620c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:00.313097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.313066 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:15:00.313302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.313271 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:15:00.321752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.321697 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv"] Nov 02 14:15:00.337199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.337168 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv"] Nov 02 14:15:00.485378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.347676 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce72d82-b7ce-44e5-8767-4249023d620c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2\" (UID: \"3ce72d82-b7ce-44e5-8767-4249023d620c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:00.485643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.357110 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:00.503006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.413043 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5076d-7f52-4d52-a99f-9113f16e668d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m\" (UID: \"0fe5076d-7f52-4d52-a99f-9113f16e668d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:00.503394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.503360 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8bx\" (UniqueName: \"kubernetes.io/projected/0fe5076d-7f52-4d52-a99f-9113f16e668d-kube-api-access-4b8bx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m\" (UID: \"0fe5076d-7f52-4d52-a99f-9113f16e668d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:00.514013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.492342 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w484\" (UniqueName: \"kubernetes.io/projected/3ce72d82-b7ce-44e5-8767-4249023d620c-kube-api-access-7w484\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2\" (UID: \"3ce72d82-b7ce-44e5-8767-4249023d620c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:00.618184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.615249 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn"] Nov 02 14:15:00.622520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.622486 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv\" (UID: \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:00.622805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.622782 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4cwc\" (UniqueName: \"kubernetes.io/projected/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-kube-api-access-s4cwc\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv\" (UID: \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:00.623058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.623035 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5076d-7f52-4d52-a99f-9113f16e668d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m\" (UID: \"0fe5076d-7f52-4d52-a99f-9113f16e668d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:00.623240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.623221 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8bx\" (UniqueName: \"kubernetes.io/projected/0fe5076d-7f52-4d52-a99f-9113f16e668d-kube-api-access-4b8bx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m\" (UID: \"0fe5076d-7f52-4d52-a99f-9113f16e668d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:00.624887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.624866 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5076d-7f52-4d52-a99f-9113f16e668d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m\" (UID: \"0fe5076d-7f52-4d52-a99f-9113f16e668d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:00.647296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.647264 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.696994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.696922 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn"] Nov 02 14:15:00.698004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.697953 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:15:00.698586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.698567 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8bx\" (UniqueName: \"kubernetes.io/projected/0fe5076d-7f52-4d52-a99f-9113f16e668d-kube-api-access-4b8bx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m\" (UID: \"0fe5076d-7f52-4d52-a99f-9113f16e668d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:00.715581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.715546 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 02 14:15:00.715734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.715604 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 02 14:15:00.724449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.724419 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.724573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.724475 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv\" (UID: \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:00.724647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.724603 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9d9\" (UniqueName: \"kubernetes.io/projected/0174298a-a6d1-497e-be27-56d96e01e223-kube-api-access-5m9d9\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.724709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.724673 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4cwc\" (UniqueName: \"kubernetes.io/projected/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-kube-api-access-s4cwc\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv\" (UID: \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:00.725064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.725040 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.725367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.725339 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv\" (UID: \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:00.783084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.782976 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4cwc\" (UniqueName: \"kubernetes.io/projected/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-kube-api-access-s4cwc\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv\" (UID: \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:00.803525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.803495 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:00.826953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.826927 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.827137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.827037 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9d9\" (UniqueName: \"kubernetes.io/projected/0174298a-a6d1-497e-be27-56d96e01e223-kube-api-access-5m9d9\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.827137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.827087 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.827745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.827723 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.829291 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.829213 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.884007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.882959 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9d9\" (UniqueName: \"kubernetes.io/projected/0174298a-a6d1-497e-be27-56d96e01e223-kube-api-access-5m9d9\") pod \"maas-api-revoke-sub-19854b08-m8dvn\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:00.899113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.899056 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:00.913215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:00.913191 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:01.083137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.064429 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:01.200109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.200067 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2"] Nov 02 14:15:01.212646 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:01.212260 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce72d82_b7ce_44e5_8767_4249023d620c.slice/crio-621edf2b13f3f9e9265c144ce5e5408474294d652c20dd6ae23022d30b6f0228 WatchSource:0}: Error finding container 621edf2b13f3f9e9265c144ce5e5408474294d652c20dd6ae23022d30b6f0228: Status 404 returned error can't find the container with id 621edf2b13f3f9e9265c144ce5e5408474294d652c20dd6ae23022d30b6f0228 Nov 02 14:15:01.214840 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:01.214702 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:15:01.332207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.331589 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:15:01.332207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.331656 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:15:01.689316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.689247 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:01.796161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.748583 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 02 14:15:01.796161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.748643 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 02 14:15:01.807192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.807063 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv"] Nov 02 14:15:01.922807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.922772 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" event={"ID":"3ce72d82-b7ce-44e5-8767-4249023d620c","Type":"ContainerStarted","Data":"c8e06845529e6e31a9b9016369bdce36dd6c07952806f08480cf7695dee7c843"} Nov 02 14:15:01.922946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.922828 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" event={"ID":"3ce72d82-b7ce-44e5-8767-4249023d620c","Type":"ContainerStarted","Data":"621edf2b13f3f9e9265c144ce5e5408474294d652c20dd6ae23022d30b6f0228"} Nov 02 14:15:01.924431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.924403 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" event={"ID":"b9d54a9f-8dd8-417d-84ed-fba446d8ac48","Type":"ContainerStarted","Data":"48c91134fbbd448a5ea9f6abb505474227571e39e2273391bbde13b86d1fea29"} Nov 02 14:15:01.975060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.975025 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn"] Nov 02 14:15:01.982299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:01.982233 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" podStartSLOduration=1.98221063 podStartE2EDuration="1.98221063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:01.974024192 +0000 UTC m=+1850.699645967" watchObservedRunningTime="2026-11-02 14:15:01.98221063 +0000 UTC m=+1850.707832401" Nov 02 14:15:02.085396 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:02.085185 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0174298a_a6d1_497e_be27_56d96e01e223.slice/crio-17e73f28df635d9da0309749ff7f670292d2076a380696e8aedb2851aeac5e96 WatchSource:0}: Error finding container 17e73f28df635d9da0309749ff7f670292d2076a380696e8aedb2851aeac5e96: Status 404 returned error can't find the container with id 17e73f28df635d9da0309749ff7f670292d2076a380696e8aedb2851aeac5e96 Nov 02 14:15:02.088226 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:02.088191 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe5076d_7f52_4d52_a99f_9113f16e668d.slice/crio-2145d9d2d96b1a0bb8a0398d0f7a829cdd9064f47a63a2f414161788c26c4482 WatchSource:0}: Error finding container 2145d9d2d96b1a0bb8a0398d0f7a829cdd9064f47a63a2f414161788c26c4482: Status 404 returned error can't find the container with id 2145d9d2d96b1a0bb8a0398d0f7a829cdd9064f47a63a2f414161788c26c4482 Nov 02 14:15:02.090530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.090350 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m"] Nov 02 14:15:02.309687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.309652 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:15:02.309852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.309705 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:15:02.715330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.715252 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 02 14:15:02.715330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.715302 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 02 14:15:02.817299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.817269 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:15:02.850748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.850710 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:02.980059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.965301 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" event={"ID":"0fe5076d-7f52-4d52-a99f-9113f16e668d","Type":"ContainerStarted","Data":"306e10931fddf3d4a5279f2c25dfe5c02c684b08794f19c5316def7043cf3160"} Nov 02 14:15:02.980059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:02.965369 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" event={"ID":"0fe5076d-7f52-4d52-a99f-9113f16e668d","Type":"ContainerStarted","Data":"2145d9d2d96b1a0bb8a0398d0f7a829cdd9064f47a63a2f414161788c26c4482"} Nov 02 14:15:03.004008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.000384 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" event={"ID":"0174298a-a6d1-497e-be27-56d96e01e223","Type":"ContainerStarted","Data":"7a1d8032bf695c872b3fa22d44d456cf9c06ff0122e6f0be4f72f82e467f8baa"} Nov 02 14:15:03.004008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.000471 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" event={"ID":"0174298a-a6d1-497e-be27-56d96e01e223","Type":"ContainerStarted","Data":"17e73f28df635d9da0309749ff7f670292d2076a380696e8aedb2851aeac5e96"} Nov 02 14:15:03.004712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.004683 5591 generic.go:356] "Generic (PLEG): container finished" podID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerID="c8e06845529e6e31a9b9016369bdce36dd6c07952806f08480cf7695dee7c843" exitCode=60 Nov 02 14:15:03.005128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.005110 5591 scope.go:122] "RemoveContainer" containerID="c8e06845529e6e31a9b9016369bdce36dd6c07952806f08480cf7695dee7c843" Nov 02 14:15:03.005281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.005243 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" event={"ID":"3ce72d82-b7ce-44e5-8767-4249023d620c","Type":"ContainerDied","Data":"c8e06845529e6e31a9b9016369bdce36dd6c07952806f08480cf7695dee7c843"} Nov 02 14:15:03.009968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.009939 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" event={"ID":"b9d54a9f-8dd8-417d-84ed-fba446d8ac48","Type":"ContainerStarted","Data":"f63e7e234949d281ce8d96ab641e94625d02676af99b274c346507bbb85f7812"} Nov 02 14:15:03.029461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.029394 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:03.094502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.094431 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" podStartSLOduration=3.094408767 podStartE2EDuration="3.094408767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.026871142 +0000 UTC m=+1851.752492913" watchObservedRunningTime="2026-11-02 14:15:03.094408767 +0000 UTC m=+1851.820030540" Nov 02 14:15:03.164663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.164593 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" podStartSLOduration=3.164568866 podStartE2EDuration="3.164568866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.097877908 +0000 UTC m=+1851.823499679" watchObservedRunningTime="2026-11-02 14:15:03.164568866 +0000 UTC m=+1851.890190731" Nov 02 14:15:03.223540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.223481 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" podStartSLOduration=3.223462561 podStartE2EDuration="3.223462561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.181294172 +0000 UTC m=+1851.906915943" watchObservedRunningTime="2026-11-02 14:15:03.223462561 +0000 UTC m=+1851.949084332" Nov 02 14:15:03.309534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.309495 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:15:03.309700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.309575 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:15:03.452652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.452581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:03.656015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.652025 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:03.804568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.804480 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:15:03.806943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.806916 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:15:03.808858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.808833 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:15:03.832888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:03.832399 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:15:04.022968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.022929 5591 generic.go:356] "Generic (PLEG): container finished" podID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerID="306e10931fddf3d4a5279f2c25dfe5c02c684b08794f19c5316def7043cf3160" exitCode=60 Nov 02 14:15:04.023230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.023067 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" event={"ID":"0fe5076d-7f52-4d52-a99f-9113f16e668d","Type":"ContainerDied","Data":"306e10931fddf3d4a5279f2c25dfe5c02c684b08794f19c5316def7043cf3160"} Nov 02 14:15:04.024038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.024015 5591 scope.go:122] "RemoveContainer" containerID="306e10931fddf3d4a5279f2c25dfe5c02c684b08794f19c5316def7043cf3160" Nov 02 14:15:04.025715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.025665 5591 generic.go:356] "Generic (PLEG): container finished" podID="0174298a-a6d1-497e-be27-56d96e01e223" containerID="7a1d8032bf695c872b3fa22d44d456cf9c06ff0122e6f0be4f72f82e467f8baa" exitCode=0 Nov 02 14:15:04.025826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.025768 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" event={"ID":"0174298a-a6d1-497e-be27-56d96e01e223","Type":"ContainerDied","Data":"7a1d8032bf695c872b3fa22d44d456cf9c06ff0122e6f0be4f72f82e467f8baa"} Nov 02 14:15:04.035795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.034332 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" event={"ID":"3ce72d82-b7ce-44e5-8767-4249023d620c","Type":"ContainerStarted","Data":"42e7a62c40eaafadd0af4ae1e3a30453948f99e597083f5527ab20316f606b49"} Nov 02 14:15:04.036890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.036866 5591 generic.go:356] "Generic (PLEG): container finished" podID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerID="f63e7e234949d281ce8d96ab641e94625d02676af99b274c346507bbb85f7812" exitCode=60 Nov 02 14:15:04.037010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.036956 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" event={"ID":"b9d54a9f-8dd8-417d-84ed-fba446d8ac48","Type":"ContainerDied","Data":"f63e7e234949d281ce8d96ab641e94625d02676af99b274c346507bbb85f7812"} Nov 02 14:15:04.037270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.037254 5591 scope.go:122] "RemoveContainer" containerID="f63e7e234949d281ce8d96ab641e94625d02676af99b274c346507bbb85f7812" Nov 02 14:15:04.313694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.313617 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:15:04.314061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.314042 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:15:04.314684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.314662 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:15:04.588065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.587952 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:04.662568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.662531 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p"] Nov 02 14:15:04.666546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.666524 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:04.684354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.684326 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-2daae4\"" Nov 02 14:15:04.686664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.686635 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p"] Nov 02 14:15:04.741457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.741391 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a181a92-b332-4b5a-bc6a-c8208b899b9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p\" (UID: \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:04.741673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.741515 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ztf\" (UniqueName: \"kubernetes.io/projected/1a181a92-b332-4b5a-bc6a-c8208b899b9c-kube-api-access-p5ztf\") pod \"payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p\" (UID: \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:04.784770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.784425 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:04.863064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.847808 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a181a92-b332-4b5a-bc6a-c8208b899b9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p\" (UID: \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:04.863064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.847912 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ztf\" (UniqueName: \"kubernetes.io/projected/1a181a92-b332-4b5a-bc6a-c8208b899b9c-kube-api-access-p5ztf\") pod \"payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p\" (UID: \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:04.863064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.849783 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a181a92-b332-4b5a-bc6a-c8208b899b9c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p\" (UID: \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:04.953013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.952137 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ztf\" (UniqueName: \"kubernetes.io/projected/1a181a92-b332-4b5a-bc6a-c8208b899b9c-kube-api-access-p5ztf\") pod \"payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p\" (UID: \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:04.997004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:04.994922 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:05.102473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.102436 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" event={"ID":"0fe5076d-7f52-4d52-a99f-9113f16e668d","Type":"ContainerStarted","Data":"6bafc52ca2dcc99f3a7dc9d16a86bf43ca7268f4ba96c147c3352453a13c37bc"} Nov 02 14:15:05.104533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.104482 5591 generic.go:356] "Generic (PLEG): container finished" podID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerID="42e7a62c40eaafadd0af4ae1e3a30453948f99e597083f5527ab20316f606b49" exitCode=60 Nov 02 14:15:05.104897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.104870 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" event={"ID":"3ce72d82-b7ce-44e5-8767-4249023d620c","Type":"ContainerDied","Data":"42e7a62c40eaafadd0af4ae1e3a30453948f99e597083f5527ab20316f606b49"} Nov 02 14:15:05.105040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.104922 5591 scope.go:122] "RemoveContainer" containerID="c8e06845529e6e31a9b9016369bdce36dd6c07952806f08480cf7695dee7c843" Nov 02 14:15:05.105283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.105257 5591 scope.go:122] "RemoveContainer" containerID="42e7a62c40eaafadd0af4ae1e3a30453948f99e597083f5527ab20316f606b49" Nov 02 14:15:05.105571 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:05.105551 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2_odh-ai-gateway-infra(3ce72d82-b7ce-44e5-8767-4249023d620c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" Nov 02 14:15:05.107409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.107384 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" event={"ID":"b9d54a9f-8dd8-417d-84ed-fba446d8ac48","Type":"ContainerStarted","Data":"a1183a32127c66dd200ea61ad499fdeb186f49873a49b75fa76140631a1b30ea"} Nov 02 14:15:05.107827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.107727 5591 scope.go:122] "RemoveContainer" containerID="a1183a32127c66dd200ea61ad499fdeb186f49873a49b75fa76140631a1b30ea" Nov 02 14:15:05.108144 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:05.108104 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv_odh-ai-gateway-infra(b9d54a9f-8dd8-417d-84ed-fba446d8ac48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" Nov 02 14:15:05.332803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.332775 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:05.408650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.408574 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0174298a-a6d1-497e-be27-56d96e01e223-kube-api-access-5m9d9\" (UniqueName: \"kubernetes.io/projected/0174298a-a6d1-497e-be27-56d96e01e223-kube-api-access-5m9d9\") pod \"0174298a-a6d1-497e-be27-56d96e01e223\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " Nov 02 14:15:05.408800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.408690 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-xks-ca\") pod \"0174298a-a6d1-497e-be27-56d96e01e223\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " Nov 02 14:15:05.408800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.408732 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-openshift-ca\") pod \"0174298a-a6d1-497e-be27-56d96e01e223\" (UID: \"0174298a-a6d1-497e-be27-56d96e01e223\") " Nov 02 14:15:05.409355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.409298 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-openshift-ca" pod "0174298a-a6d1-497e-be27-56d96e01e223" (UID: "0174298a-a6d1-497e-be27-56d96e01e223"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:05.411052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.410954 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-xks-ca" pod "0174298a-a6d1-497e-be27-56d96e01e223" (UID: "0174298a-a6d1-497e-be27-56d96e01e223"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:15:05.411213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.411191 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0174298a-a6d1-497e-be27-56d96e01e223-kube-api-access-5m9d9" pod "0174298a-a6d1-497e-be27-56d96e01e223" (UID: "0174298a-a6d1-497e-be27-56d96e01e223"). InnerVolumeSpecName "kube-api-access-5m9d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:05.415566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.415541 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p"] Nov 02 14:15:05.416797 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:05.416772 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a181a92_b332_4b5a_bc6a_c8208b899b9c.slice/crio-9ca011cca25877dfb3b4a6d71904c6ac753d931f56a625f3aa72237e09f0ff0b WatchSource:0}: Error finding container 9ca011cca25877dfb3b4a6d71904c6ac753d931f56a625f3aa72237e09f0ff0b: Status 404 returned error can't find the container with id 9ca011cca25877dfb3b4a6d71904c6ac753d931f56a625f3aa72237e09f0ff0b Nov 02 14:15:05.510968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.510937 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9d9\" (UniqueName: \"kubernetes.io/projected/0174298a-a6d1-497e-be27-56d96e01e223-kube-api-access-5m9d9\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:05.510968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.510964 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:05.511148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.510974 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0174298a-a6d1-497e-be27-56d96e01e223-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:05.615072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:05.615032 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:06.135276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.135234 5591 generic.go:356] "Generic (PLEG): container finished" podID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerID="6bafc52ca2dcc99f3a7dc9d16a86bf43ca7268f4ba96c147c3352453a13c37bc" exitCode=60 Nov 02 14:15:06.135840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.135321 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" event={"ID":"0fe5076d-7f52-4d52-a99f-9113f16e668d","Type":"ContainerDied","Data":"6bafc52ca2dcc99f3a7dc9d16a86bf43ca7268f4ba96c147c3352453a13c37bc"} Nov 02 14:15:06.135840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.135394 5591 scope.go:122] "RemoveContainer" containerID="306e10931fddf3d4a5279f2c25dfe5c02c684b08794f19c5316def7043cf3160" Nov 02 14:15:06.135840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.135672 5591 scope.go:122] "RemoveContainer" containerID="6bafc52ca2dcc99f3a7dc9d16a86bf43ca7268f4ba96c147c3352453a13c37bc" Nov 02 14:15:06.136119 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:06.136095 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m_odh-ai-gateway-infra(0fe5076d-7f52-4d52-a99f-9113f16e668d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" Nov 02 14:15:06.137304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.136876 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" Nov 02 14:15:06.137304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.136894 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn" event={"ID":"0174298a-a6d1-497e-be27-56d96e01e223","Type":"ContainerDied","Data":"17e73f28df635d9da0309749ff7f670292d2076a380696e8aedb2851aeac5e96"} Nov 02 14:15:06.137304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.136931 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17e73f28df635d9da0309749ff7f670292d2076a380696e8aedb2851aeac5e96" Nov 02 14:15:06.138902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.138786 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" event={"ID":"1a181a92-b332-4b5a-bc6a-c8208b899b9c","Type":"ContainerStarted","Data":"91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a"} Nov 02 14:15:06.138902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.138822 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" event={"ID":"1a181a92-b332-4b5a-bc6a-c8208b899b9c","Type":"ContainerStarted","Data":"9ca011cca25877dfb3b4a6d71904c6ac753d931f56a625f3aa72237e09f0ff0b"} Nov 02 14:15:06.138902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.138854 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:06.141664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.141637 5591 scope.go:122] "RemoveContainer" containerID="42e7a62c40eaafadd0af4ae1e3a30453948f99e597083f5527ab20316f606b49" Nov 02 14:15:06.141944 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:06.141921 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2_odh-ai-gateway-infra(3ce72d82-b7ce-44e5-8767-4249023d620c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" Nov 02 14:15:06.143455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.143432 5591 generic.go:356] "Generic (PLEG): container finished" podID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerID="a1183a32127c66dd200ea61ad499fdeb186f49873a49b75fa76140631a1b30ea" exitCode=60 Nov 02 14:15:06.143551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.143516 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" event={"ID":"b9d54a9f-8dd8-417d-84ed-fba446d8ac48","Type":"ContainerDied","Data":"a1183a32127c66dd200ea61ad499fdeb186f49873a49b75fa76140631a1b30ea"} Nov 02 14:15:06.143809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.143795 5591 scope.go:122] "RemoveContainer" containerID="a1183a32127c66dd200ea61ad499fdeb186f49873a49b75fa76140631a1b30ea" Nov 02 14:15:06.144040 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:06.144018 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv_odh-ai-gateway-infra(b9d54a9f-8dd8-417d-84ed-fba446d8ac48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" Nov 02 14:15:06.148978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.148958 5591 scope.go:122] "RemoveContainer" containerID="f63e7e234949d281ce8d96ab641e94625d02676af99b274c346507bbb85f7812" Nov 02 14:15:06.486602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:06.486532 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" podStartSLOduration=2.486511095 podStartE2EDuration="2.486511095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:06.343881918 +0000 UTC m=+1855.069503689" watchObservedRunningTime="2026-11-02 14:15:06.486511095 +0000 UTC m=+1855.212132866" Nov 02 14:15:07.152414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:07.152376 5591 scope.go:122] "RemoveContainer" containerID="6bafc52ca2dcc99f3a7dc9d16a86bf43ca7268f4ba96c147c3352453a13c37bc" Nov 02 14:15:07.152905 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:07.152760 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m_odh-ai-gateway-infra(0fe5076d-7f52-4d52-a99f-9113f16e668d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" Nov 02 14:15:07.154175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:07.154149 5591 scope.go:122] "RemoveContainer" containerID="a1183a32127c66dd200ea61ad499fdeb186f49873a49b75fa76140631a1b30ea" Nov 02 14:15:07.154452 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:07.154430 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv_odh-ai-gateway-infra(b9d54a9f-8dd8-417d-84ed-fba446d8ac48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" Nov 02 14:15:07.568978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:07.564735 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:09.326651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:09.326613 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:10.945910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:10.945874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:12.660689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:12.660645 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:14.619944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:14.619906 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:14.788090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:14.788057 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:15.029284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:15.029248 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:15.429022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:15.428970 5591 scope.go:122] "RemoveContainer" containerID="42e7a62c40eaafadd0af4ae1e3a30453948f99e597083f5527ab20316f606b49" Nov 02 14:15:16.211879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:16.211842 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" event={"ID":"3ce72d82-b7ce-44e5-8767-4249023d620c","Type":"ContainerStarted","Data":"6f88c5542f43685eb74b765fb60b58fa831a3013fb22aa8af94861e6e7e7fe33"} Nov 02 14:15:16.423013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:16.422962 5591 scope.go:122] "RemoveContainer" containerID="6bafc52ca2dcc99f3a7dc9d16a86bf43ca7268f4ba96c147c3352453a13c37bc" Nov 02 14:15:16.423316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:16.423286 5591 scope.go:122] "RemoveContainer" containerID="a1183a32127c66dd200ea61ad499fdeb186f49873a49b75fa76140631a1b30ea" Nov 02 14:15:16.543945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:16.543900 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:17.160628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.160593 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:15:17.221224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.221190 5591 generic.go:356] "Generic (PLEG): container finished" podID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerID="6f88c5542f43685eb74b765fb60b58fa831a3013fb22aa8af94861e6e7e7fe33" exitCode=60 Nov 02 14:15:17.221701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.221264 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" event={"ID":"3ce72d82-b7ce-44e5-8767-4249023d620c","Type":"ContainerDied","Data":"6f88c5542f43685eb74b765fb60b58fa831a3013fb22aa8af94861e6e7e7fe33"} Nov 02 14:15:17.221701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.221326 5591 scope.go:122] "RemoveContainer" containerID="42e7a62c40eaafadd0af4ae1e3a30453948f99e597083f5527ab20316f606b49" Nov 02 14:15:17.222197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.222175 5591 scope.go:122] "RemoveContainer" containerID="6f88c5542f43685eb74b765fb60b58fa831a3013fb22aa8af94861e6e7e7fe33" Nov 02 14:15:17.222745 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:17.222717 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2_odh-ai-gateway-infra(3ce72d82-b7ce-44e5-8767-4249023d620c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" Nov 02 14:15:17.223611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.223582 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" event={"ID":"b9d54a9f-8dd8-417d-84ed-fba446d8ac48","Type":"ContainerStarted","Data":"3d94f6a6e7d9252e966b004620b23569ce0d840b875eea5655ac3adfbcd74c43"} Nov 02 14:15:17.225953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.225932 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" event={"ID":"0fe5076d-7f52-4d52-a99f-9113f16e668d","Type":"ContainerStarted","Data":"382d2c5409e922c930ed8c80519835a13caedfb31d31982e987c383ba1715c76"} Nov 02 14:15:17.299480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.299445 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2"] Nov 02 14:15:17.793825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:17.790628 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:18.234765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.234725 5591 generic.go:356] "Generic (PLEG): container finished" podID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerID="382d2c5409e922c930ed8c80519835a13caedfb31d31982e987c383ba1715c76" exitCode=60 Nov 02 14:15:18.235355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.234924 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" event={"ID":"0fe5076d-7f52-4d52-a99f-9113f16e668d","Type":"ContainerDied","Data":"382d2c5409e922c930ed8c80519835a13caedfb31d31982e987c383ba1715c76"} Nov 02 14:15:18.235504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.235404 5591 scope.go:122] "RemoveContainer" containerID="6bafc52ca2dcc99f3a7dc9d16a86bf43ca7268f4ba96c147c3352453a13c37bc" Nov 02 14:15:18.235871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.235851 5591 scope.go:122] "RemoveContainer" containerID="382d2c5409e922c930ed8c80519835a13caedfb31d31982e987c383ba1715c76" Nov 02 14:15:18.236332 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:18.236304 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m_odh-ai-gateway-infra(0fe5076d-7f52-4d52-a99f-9113f16e668d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" Nov 02 14:15:18.240226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.240198 5591 generic.go:356] "Generic (PLEG): container finished" podID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerID="3d94f6a6e7d9252e966b004620b23569ce0d840b875eea5655ac3adfbcd74c43" exitCode=60 Nov 02 14:15:18.240388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.240335 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" event={"ID":"b9d54a9f-8dd8-417d-84ed-fba446d8ac48","Type":"ContainerDied","Data":"3d94f6a6e7d9252e966b004620b23569ce0d840b875eea5655ac3adfbcd74c43"} Nov 02 14:15:18.240702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.240684 5591 scope.go:122] "RemoveContainer" containerID="3d94f6a6e7d9252e966b004620b23569ce0d840b875eea5655ac3adfbcd74c43" Nov 02 14:15:18.241017 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:18.240976 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv_odh-ai-gateway-infra(b9d54a9f-8dd8-417d-84ed-fba446d8ac48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" Nov 02 14:15:18.250527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.250503 5591 scope.go:122] "RemoveContainer" containerID="a1183a32127c66dd200ea61ad499fdeb186f49873a49b75fa76140631a1b30ea" Nov 02 14:15:18.315314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.315267 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:18.425007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.417978 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv"] Nov 02 14:15:18.440950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.440173 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m"] Nov 02 14:15:18.497637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.497570 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:18.559554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.559514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:18.612949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.612910 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce72d82-b7ce-44e5-8767-4249023d620c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce72d82-b7ce-44e5-8767-4249023d620c-maas-api-service-ca\") pod \"3ce72d82-b7ce-44e5-8767-4249023d620c\" (UID: \"3ce72d82-b7ce-44e5-8767-4249023d620c\") " Nov 02 14:15:18.613184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.613081 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce72d82-b7ce-44e5-8767-4249023d620c-kube-api-access-7w484\" (UniqueName: \"kubernetes.io/projected/3ce72d82-b7ce-44e5-8767-4249023d620c-kube-api-access-7w484\") pod \"3ce72d82-b7ce-44e5-8767-4249023d620c\" (UID: \"3ce72d82-b7ce-44e5-8767-4249023d620c\") " Nov 02 14:15:18.613329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.613303 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce72d82-b7ce-44e5-8767-4249023d620c-maas-api-service-ca" pod "3ce72d82-b7ce-44e5-8767-4249023d620c" (UID: "3ce72d82-b7ce-44e5-8767-4249023d620c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:18.613638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.613621 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce72d82-b7ce-44e5-8767-4249023d620c-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:18.615186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.615165 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce72d82-b7ce-44e5-8767-4249023d620c-kube-api-access-7w484" pod "3ce72d82-b7ce-44e5-8767-4249023d620c" (UID: "3ce72d82-b7ce-44e5-8767-4249023d620c"). InnerVolumeSpecName "kube-api-access-7w484". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:18.715435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.715401 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w484\" (UniqueName: \"kubernetes.io/projected/3ce72d82-b7ce-44e5-8767-4249023d620c-kube-api-access-7w484\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:18.867093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:18.865370 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:19.247550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.247520 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" Nov 02 14:15:19.248048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.247524 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2" event={"ID":"3ce72d82-b7ce-44e5-8767-4249023d620c","Type":"ContainerDied","Data":"621edf2b13f3f9e9265c144ce5e5408474294d652c20dd6ae23022d30b6f0228"} Nov 02 14:15:19.248048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.247669 5591 scope.go:122] "RemoveContainer" containerID="6f88c5542f43685eb74b765fb60b58fa831a3013fb22aa8af94861e6e7e7fe33" Nov 02 14:15:19.359249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.359018 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2"] Nov 02 14:15:19.364244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.364214 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893815-572z2"] Nov 02 14:15:19.365241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.364914 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:19.437440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.437407 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" path="/var/lib/kubelet/pods/3ce72d82-b7ce-44e5-8767-4249023d620c/volumes" Nov 02 14:15:19.508338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.508266 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:19.511633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.511608 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:19.584300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.584262 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe5076d-7f52-4d52-a99f-9113f16e668d-kube-api-access-4b8bx\" (UniqueName: \"kubernetes.io/projected/0fe5076d-7f52-4d52-a99f-9113f16e668d-kube-api-access-4b8bx\") pod \"0fe5076d-7f52-4d52-a99f-9113f16e668d\" (UID: \"0fe5076d-7f52-4d52-a99f-9113f16e668d\") " Nov 02 14:15:19.584492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.584330 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-maas-api-service-ca\") pod \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\" (UID: \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\") " Nov 02 14:15:19.584492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.584391 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe5076d-7f52-4d52-a99f-9113f16e668d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5076d-7f52-4d52-a99f-9113f16e668d-maas-api-service-ca\") pod \"0fe5076d-7f52-4d52-a99f-9113f16e668d\" (UID: \"0fe5076d-7f52-4d52-a99f-9113f16e668d\") " Nov 02 14:15:19.584628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.584499 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-kube-api-access-s4cwc\" (UniqueName: \"kubernetes.io/projected/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-kube-api-access-s4cwc\") pod \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\" (UID: \"b9d54a9f-8dd8-417d-84ed-fba446d8ac48\") " Nov 02 14:15:19.584785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.584758 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-maas-api-service-ca" pod "b9d54a9f-8dd8-417d-84ed-fba446d8ac48" (UID: "b9d54a9f-8dd8-417d-84ed-fba446d8ac48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:19.584904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.584850 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe5076d-7f52-4d52-a99f-9113f16e668d-maas-api-service-ca" pod "0fe5076d-7f52-4d52-a99f-9113f16e668d" (UID: "0fe5076d-7f52-4d52-a99f-9113f16e668d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:19.585031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.585015 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:19.585076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.585036 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5076d-7f52-4d52-a99f-9113f16e668d-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:19.586718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.586694 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-kube-api-access-s4cwc" pod "b9d54a9f-8dd8-417d-84ed-fba446d8ac48" (UID: "b9d54a9f-8dd8-417d-84ed-fba446d8ac48"). InnerVolumeSpecName "kube-api-access-s4cwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:19.586840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.586734 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe5076d-7f52-4d52-a99f-9113f16e668d-kube-api-access-4b8bx" pod "0fe5076d-7f52-4d52-a99f-9113f16e668d" (UID: "0fe5076d-7f52-4d52-a99f-9113f16e668d"). InnerVolumeSpecName "kube-api-access-4b8bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:19.686930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.686891 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4cwc\" (UniqueName: \"kubernetes.io/projected/b9d54a9f-8dd8-417d-84ed-fba446d8ac48-kube-api-access-s4cwc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:19.686930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:19.686927 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8bx\" (UniqueName: \"kubernetes.io/projected/0fe5076d-7f52-4d52-a99f-9113f16e668d-kube-api-access-4b8bx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:15:20.261631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.261596 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" Nov 02 14:15:20.262113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.261629 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m" event={"ID":"0fe5076d-7f52-4d52-a99f-9113f16e668d","Type":"ContainerDied","Data":"2145d9d2d96b1a0bb8a0398d0f7a829cdd9064f47a63a2f414161788c26c4482"} Nov 02 14:15:20.262113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.261691 5591 scope.go:122] "RemoveContainer" containerID="382d2c5409e922c930ed8c80519835a13caedfb31d31982e987c383ba1715c76" Nov 02 14:15:20.264968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.264934 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" Nov 02 14:15:20.264968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.264954 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv" event={"ID":"b9d54a9f-8dd8-417d-84ed-fba446d8ac48","Type":"ContainerDied","Data":"48c91134fbbd448a5ea9f6abb505474227571e39e2273391bbde13b86d1fea29"} Nov 02 14:15:20.274560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.274542 5591 scope.go:122] "RemoveContainer" containerID="3d94f6a6e7d9252e966b004620b23569ce0d840b875eea5655ac3adfbcd74c43" Nov 02 14:15:20.318800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.318766 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv"] Nov 02 14:15:20.331527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.331498 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893815-8vkwv"] Nov 02 14:15:20.449341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.435693 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:20.449341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.436015 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m"] Nov 02 14:15:20.453249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:20.453215 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893815-rt98m"] Nov 02 14:15:21.439587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:21.439546 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" path="/var/lib/kubelet/pods/0fe5076d-7f52-4d52-a99f-9113f16e668d/volumes" Nov 02 14:15:21.440103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:21.440020 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" path="/var/lib/kubelet/pods/b9d54a9f-8dd8-417d-84ed-fba446d8ac48/volumes" Nov 02 14:15:22.215016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:22.204735 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:22.666417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:22.666375 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:22.849343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:22.849308 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:23.557617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:23.557582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:24.237085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.235854 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:24.411908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.411876 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:24.490789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.490708 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd"] Nov 02 14:15:24.491417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491395 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:24.491417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491416 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491438 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491445 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491459 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491467 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491488 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491495 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491507 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0174298a-a6d1-497e-be27-56d96e01e223" containerName="revoke-keys" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491514 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0174298a-a6d1-497e-be27-56d96e01e223" containerName="revoke-keys" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491543 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491550 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491578 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:24.491593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491585 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:24.492213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491716 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:24.492213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491732 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:24.492213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491741 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:24.492213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491751 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:24.492213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491759 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0174298a-a6d1-497e-be27-56d96e01e223" containerName="revoke-keys" Nov 02 14:15:24.492213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491773 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:24.492213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491781 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:24.492213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.491792 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:24.495522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.495501 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:24.526070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.526041 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-0d1ca7\"" Nov 02 14:15:24.526276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.526257 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-0d1ca7-dockercfg-hn2t9\"" Nov 02 14:15:24.529226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.529202 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd"] Nov 02 14:15:24.642016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.640937 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:24.799072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.798659 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd\" (UID: \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:24.799376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.799349 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fc5l\" (UniqueName: \"kubernetes.io/projected/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-kube-api-access-8fc5l\") pod \"payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd\" (UID: \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:24.900997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.900951 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd\" (UID: \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:24.901190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.901096 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fc5l\" (UniqueName: \"kubernetes.io/projected/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-kube-api-access-8fc5l\") pod \"payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd\" (UID: \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:24.901744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.901721 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd\" (UID: \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:24.925237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:24.925207 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fc5l\" (UniqueName: \"kubernetes.io/projected/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-kube-api-access-8fc5l\") pod \"payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd\" (UID: \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:25.108219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.108133 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:25.411926 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:25.411894 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3d1492_01c4_453d_bcb9_4a18c0fbe6d7.slice/crio-dd2b85a4dc2acfae5e1713472dc29612829c07be9cd76b9a4adfd68e7d50beaf WatchSource:0}: Error finding container dd2b85a4dc2acfae5e1713472dc29612829c07be9cd76b9a4adfd68e7d50beaf: Status 404 returned error can't find the container with id dd2b85a4dc2acfae5e1713472dc29612829c07be9cd76b9a4adfd68e7d50beaf Nov 02 14:15:25.412553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.412522 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd"] Nov 02 14:15:25.976536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.976501 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5"] Nov 02 14:15:25.977135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.977120 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:25.977187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.977136 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce72d82-b7ce-44e5-8767-4249023d620c" containerName="cleanup" Nov 02 14:15:25.977187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.977171 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:25.977187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.977176 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:25.977279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.977191 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:25.977279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.977196 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:25.977279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.977277 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe5076d-7f52-4d52-a99f-9113f16e668d" containerName="cleanup" Nov 02 14:15:25.977372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.977294 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d54a9f-8dd8-417d-84ed-fba446d8ac48" containerName="cleanup" Nov 02 14:15:25.979691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:25.979673 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.001100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.001064 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-2daae4-serving-cert\"" Nov 02 14:15:26.004506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.004480 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5"] Nov 02 14:15:26.016832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.016796 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2223ad11-a345-472c-8254-879383227f0b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.017039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.016971 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zvh\" (UniqueName: \"kubernetes.io/projected/2223ad11-a345-472c-8254-879383227f0b-kube-api-access-t7zvh\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.017189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.017149 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.017334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.017205 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.107189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.107125 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:26.118338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.118309 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.118480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.118352 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.118480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.118474 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2223ad11-a345-472c-8254-879383227f0b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.118585 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:26.118514 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-serving-cert: secret "maas-api-e2e-shared-b-w0-2daae4-serving-cert" not found Nov 02 14:15:26.118585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.118525 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zvh\" (UniqueName: \"kubernetes.io/projected/2223ad11-a345-472c-8254-879383227f0b-kube-api-access-t7zvh\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.118585 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:26.118575 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls podName:2223ad11-a345-472c-8254-879383227f0b nodeName:}" failed. No retries permitted until 2026-11-02 14:15:26.61856059 +0000 UTC m=+1875.344182338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls") pod "maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" (UID: "2223ad11-a345-472c-8254-879383227f0b") : secret "maas-api-e2e-shared-b-w0-2daae4-serving-cert" not found Nov 02 14:15:26.119109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.119090 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2223ad11-a345-472c-8254-879383227f0b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.121040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.121013 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.179061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.178969 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zvh\" (UniqueName: \"kubernetes.io/projected/2223ad11-a345-472c-8254-879383227f0b-kube-api-access-t7zvh\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.273277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.273186 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:26.308263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.308227 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" event={"ID":"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7","Type":"ContainerStarted","Data":"69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3"} Nov 02 14:15:26.308423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.308291 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" event={"ID":"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7","Type":"ContainerStarted","Data":"dd2b85a4dc2acfae5e1713472dc29612829c07be9cd76b9a4adfd68e7d50beaf"} Nov 02 14:15:26.308423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.308319 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:26.352386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.352331 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" podStartSLOduration=2.352316017 podStartE2EDuration="2.352316017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:26.350685873 +0000 UTC m=+1875.076307625" watchObservedRunningTime="2026-11-02 14:15:26.352316017 +0000 UTC m=+1875.077937787" Nov 02 14:15:26.456753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.456719 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:26.627937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.627840 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.630406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.630383 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:26.893923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:26.893882 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:27.127663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:27.127627 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:15:27.259008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:27.247527 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5"] Nov 02 14:15:27.438431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:27.438393 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" event={"ID":"2223ad11-a345-472c-8254-879383227f0b","Type":"ContainerStarted","Data":"6c964b3880c5f96056d60cc096d31402fccdf6a335201dd535be52c6ff6c1a8a"} Nov 02 14:15:28.209008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:28.206841 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:28.438240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:28.438198 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" event={"ID":"2223ad11-a345-472c-8254-879383227f0b","Type":"ContainerStarted","Data":"66a071d43f243e2748658e2d9521a298e8901cdd4689e92830c59a49459a6993"} Nov 02 14:15:28.438446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:28.438341 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:28.499758 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:28.499645 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" podStartSLOduration=3.132753519 podStartE2EDuration="3.499631883s" totalImagesPullingTime="366.878364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:28.49671369 +0000 UTC m=+1877.222335461" watchObservedRunningTime="2026-11-02 14:15:28.499631883 +0000 UTC m=+1877.225253654" Nov 02 14:15:30.073012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:30.069443 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:31.086116 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:15:31.086074 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:15:31.801063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:31.795378 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:31.965106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:31.965067 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:32.742916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:32.742870 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:32.997688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:32.997600 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:33.864494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:33.864456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:34.465897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:34.465867 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:15:35.343660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.343605 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:35.592625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.592554 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:35.658534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.658484 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd"] Nov 02 14:15:35.670384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.670213 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.689188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.689155 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd"] Nov 02 14:15:35.711320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.702394 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.711641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.711609 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.711959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.711929 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.712128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.712083 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.712188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.712139 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxv9b\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-kube-api-access-hxv9b\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.712188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.712172 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.712292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.712259 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.712439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.712359 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.712575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.712555 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.814456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814414 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.814654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814474 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.814654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814520 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxv9b\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-kube-api-access-hxv9b\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.814654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814560 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.814654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814598 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.814882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814661 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.814882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814830 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.814882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814872 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.815080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814902 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.815080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.814927 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.815396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.815366 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.815761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.815711 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.815871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.815775 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.815931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.815907 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.817709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.817686 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.818024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.817962 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.840060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.840029 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxv9b\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-kube-api-access-hxv9b\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.840742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.840715 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:35.980029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.979928 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:35.993661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:35.993629 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:36.175564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.175511 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:36.331069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.331036 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd"] Nov 02 14:15:36.331662 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:36.331630 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9a4390_351a_4886_90ee_b2ca1ad9d931.slice/crio-d3563655df1b606aadbba1c8cc4d0dcec0440ac7370ccbaf16ae8960281e3fbe WatchSource:0}: Error finding container d3563655df1b606aadbba1c8cc4d0dcec0440ac7370ccbaf16ae8960281e3fbe: Status 404 returned error can't find the container with id d3563655df1b606aadbba1c8cc4d0dcec0440ac7370ccbaf16ae8960281e3fbe Nov 02 14:15:36.334442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.334411 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:36.334530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.334477 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:36.334530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.334505 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:36.550005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.549919 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" event={"ID":"fd9a4390-351a-4886-90ee-b2ca1ad9d931","Type":"ContainerStarted","Data":"f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9"} Nov 02 14:15:36.550005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.550003 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" event={"ID":"fd9a4390-351a-4886-90ee-b2ca1ad9d931","Type":"ContainerStarted","Data":"d3563655df1b606aadbba1c8cc4d0dcec0440ac7370ccbaf16ae8960281e3fbe"} Nov 02 14:15:36.608322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.608210 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" podStartSLOduration=1.608195527 podStartE2EDuration="1.608195527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:36.604362 +0000 UTC m=+1885.329983773" watchObservedRunningTime="2026-11-02 14:15:36.608195527 +0000 UTC m=+1885.333817296" Nov 02 14:15:36.994762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:36.994727 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:37.468316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.468283 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:15:37.663943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.663589 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:37.769415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.769327 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn"] Nov 02 14:15:37.774776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.774751 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:37.791235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.791202 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5cbbfb-dockercfg-mp4jg\"" Nov 02 14:15:37.791445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.791297 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5cbbfb\"" Nov 02 14:15:37.791769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.791741 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn"] Nov 02 14:15:37.949008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.881381 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbdl\" (UniqueName: \"kubernetes.io/projected/37d3b616-82de-42fd-9990-97fef65c2781-kube-api-access-fvbdl\") pod \"payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn\" (UID: \"37d3b616-82de-42fd-9990-97fef65c2781\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:37.957007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.953376 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d3b616-82de-42fd-9990-97fef65c2781-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn\" (UID: \"37d3b616-82de-42fd-9990-97fef65c2781\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:37.965025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:37.931561 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:38.024058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.009317 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:15:38.024058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.009385 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:15:38.080008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.074156 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d3b616-82de-42fd-9990-97fef65c2781-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn\" (UID: \"37d3b616-82de-42fd-9990-97fef65c2781\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:38.080008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.074295 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbdl\" (UniqueName: \"kubernetes.io/projected/37d3b616-82de-42fd-9990-97fef65c2781-kube-api-access-fvbdl\") pod \"payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn\" (UID: \"37d3b616-82de-42fd-9990-97fef65c2781\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:38.080008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.075687 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d3b616-82de-42fd-9990-97fef65c2781-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn\" (UID: \"37d3b616-82de-42fd-9990-97fef65c2781\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:38.164011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.161203 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbdl\" (UniqueName: \"kubernetes.io/projected/37d3b616-82de-42fd-9990-97fef65c2781-kube-api-access-fvbdl\") pod \"payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn\" (UID: \"37d3b616-82de-42fd-9990-97fef65c2781\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:38.395615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.395579 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:38.764270 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:38.764213 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d3b616_82de_42fd_9990_97fef65c2781.slice/crio-22661ff7aef234ec6edc01ad4059f2cbd89f0104a912588b185298907d3457f8 WatchSource:0}: Error finding container 22661ff7aef234ec6edc01ad4059f2cbd89f0104a912588b185298907d3457f8: Status 404 returned error can't find the container with id 22661ff7aef234ec6edc01ad4059f2cbd89f0104a912588b185298907d3457f8 Nov 02 14:15:38.765369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.765337 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn"] Nov 02 14:15:38.967604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.966134 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": read tcp 10.133.0.2:44188->10.133.0.164:15021: read: connection reset by peer" start-of-body= Nov 02 14:15:38.967604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.966214 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": read tcp 10.133.0.2:44188->10.133.0.164:15021: read: connection reset by peer" Nov 02 14:15:38.994344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.994310 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:15:38.994465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:38.994364 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:15:39.512315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.512227 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:39.593564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.593530 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" event={"ID":"37d3b616-82de-42fd-9990-97fef65c2781","Type":"ContainerStarted","Data":"78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862"} Nov 02 14:15:39.593783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.593588 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" event={"ID":"37d3b616-82de-42fd-9990-97fef65c2781","Type":"ContainerStarted","Data":"22661ff7aef234ec6edc01ad4059f2cbd89f0104a912588b185298907d3457f8"} Nov 02 14:15:39.593783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.593641 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:39.595286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.595259 5591 generic.go:356] "Generic (PLEG): container finished" podID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerID="de630999105654b44dd9a3973428dc8f23ca70d8fbd7ab7336ffcc8129536ee0" exitCode=0 Nov 02 14:15:39.595415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.595332 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" event={"ID":"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7","Type":"ContainerDied","Data":"de630999105654b44dd9a3973428dc8f23ca70d8fbd7ab7336ffcc8129536ee0"} Nov 02 14:15:39.595689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.595675 5591 scope.go:122] "RemoveContainer" containerID="de630999105654b44dd9a3973428dc8f23ca70d8fbd7ab7336ffcc8129536ee0" Nov 02 14:15:39.597091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.597066 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:39.597166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.597130 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:39.597166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.597161 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:39.738005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.733997 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" podStartSLOduration=2.733934438 podStartE2EDuration="2.733934438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:39.645284136 +0000 UTC m=+1888.370905907" watchObservedRunningTime="2026-11-02 14:15:39.733934438 +0000 UTC m=+1888.459556209" Nov 02 14:15:40.005902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.995942 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:15:40.005902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.996031 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:15:40.005902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:39.997053 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:40.367255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.367173 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw"] Nov 02 14:15:40.371598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.371577 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.409072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.409041 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4c2811-openshift-default-dockercfg-pqrrz\"" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416342 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416492 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416537 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416638 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416684 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdcpf\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-kube-api-access-qdcpf\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416727 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2574d71-5766-4983-9fbd-41c3103170e4-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416758 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416810 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2574d71-5766-4983-9fbd-41c3103170e4-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.416858 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.421542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.417069 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw"] Nov 02 14:15:40.518512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.518472 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.518726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.518700 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.519141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.519111 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.519270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.519206 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.519355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.519272 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdcpf\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-kube-api-access-qdcpf\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.519355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.519325 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2574d71-5766-4983-9fbd-41c3103170e4-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.519468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.519366 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.519468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.519433 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2574d71-5766-4983-9fbd-41c3103170e4-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.519653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.519475 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.519653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.519520 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.520327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.520301 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2574d71-5766-4983-9fbd-41c3103170e4-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.520438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.520348 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.520713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.520688 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.520873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.520857 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.522222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.522201 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.522347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.522326 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2574d71-5766-4983-9fbd-41c3103170e4-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.583018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.582182 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdcpf\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-kube-api-access-qdcpf\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.583018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.582718 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.604030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.603661 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" event={"ID":"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7","Type":"ContainerStarted","Data":"915ee59361dd4e6c600622f41779911aaeda2fc9514c3172e9005e175fd25119"} Nov 02 14:15:40.688035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.687252 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:40.909778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.909746 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:15:40.910671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.910640 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 14:15:40.910817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.910699 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 14:15:40.994551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.994458 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:15:40.994551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:40.994519 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:15:41.091453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.091422 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw"] Nov 02 14:15:41.091882 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:41.091858 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2574d71_5766_4983_9fbd_41c3103170e4.slice/crio-343ba3091103a89eeb940c909cdb9dc91134d189dabd8f8c9bda06a351704500 WatchSource:0}: Error finding container 343ba3091103a89eeb940c909cdb9dc91134d189dabd8f8c9bda06a351704500: Status 404 returned error can't find the container with id 343ba3091103a89eeb940c909cdb9dc91134d189dabd8f8c9bda06a351704500 Nov 02 14:15:41.094631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.094599 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:41.094713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.094662 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:41.094713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.094693 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:15:41.650006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.648245 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" event={"ID":"d2574d71-5766-4983-9fbd-41c3103170e4","Type":"ContainerStarted","Data":"5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565"} Nov 02 14:15:41.650006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.648307 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" event={"ID":"d2574d71-5766-4983-9fbd-41c3103170e4","Type":"ContainerStarted","Data":"343ba3091103a89eeb940c909cdb9dc91134d189dabd8f8c9bda06a351704500"} Nov 02 14:15:41.692006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.688316 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:41.913142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.913035 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 14:15:41.913396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.913369 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 14:15:41.996879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.996840 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:15:41.997181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:41.996902 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:15:42.085007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:42.083492 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:42.714011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:42.708688 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 14:15:42.714011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:42.708746 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 14:15:42.910199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:42.910166 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 14:15:42.910387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:42.910221 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 14:15:42.996541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:42.994669 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:15:42.996541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:42.994729 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:15:43.689661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:43.689611 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 14:15:43.689859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:43.689679 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 14:15:43.910727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:43.910689 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 14:15:43.911129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:43.910750 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 14:15:43.998608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:43.998532 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:43.999230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:43.999202 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:44.000107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:44.000086 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:15:44.080732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:44.080657 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" podStartSLOduration=4.080625144 podStartE2EDuration="4.080625144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:41.746396343 +0000 UTC m=+1890.472018116" watchObservedRunningTime="2026-11-02 14:15:44.080625144 +0000 UTC m=+1892.806246914" Nov 02 14:15:44.290063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:44.280621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:44.689068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:44.689023 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 14:15:44.689320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:44.689102 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 14:15:44.910590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:44.910218 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 14:15:44.910590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:44.910271 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 14:15:45.189008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:45.185710 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:45.689607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:45.689569 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 14:15:45.689887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:45.689859 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 14:15:45.910566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:45.910532 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 14:15:45.910745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:45.910582 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 14:15:45.996548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:45.996464 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:46.689379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:46.689334 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 14:15:46.689867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:46.689413 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 14:15:46.916834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:46.916801 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:15:46.917258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:46.917228 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:15:46.918112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:46.918089 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:15:47.689292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.689259 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 14:15:47.689482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.689311 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 14:15:47.886328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.886294 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6"] Nov 02 14:15:47.891075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.891048 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:47.931880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.931824 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-0d1ca7-serving-cert\"" Nov 02 14:15:47.931880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.931865 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:47.932179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.932034 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:47.932247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.932232 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppv84\" (UniqueName: \"kubernetes.io/projected/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-kube-api-access-ppv84\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:47.932421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.932318 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:47.937759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:47.937716 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6"] Nov 02 14:15:48.033657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.033560 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.033657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.033616 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.033903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.033691 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.033903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.033832 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv84\" (UniqueName: \"kubernetes.io/projected/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-kube-api-access-ppv84\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.034419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.034390 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.036272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.036249 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.036412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.036393 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.042240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.042217 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:48.058495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.058467 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppv84\" (UniqueName: \"kubernetes.io/projected/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-kube-api-access-ppv84\") pod \"maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.215852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.215804 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:48.216323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.216298 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:48.537509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.537478 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6"] Nov 02 14:15:48.538600 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:15:48.538572 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e862b91_b6c8_44b6_b8d3_ca45d890af9d.slice/crio-7dbb8fbd3842df8f698fc0ba186f62a7fb9bc14a09cfe717adc5bbd17d99fcb5 WatchSource:0}: Error finding container 7dbb8fbd3842df8f698fc0ba186f62a7fb9bc14a09cfe717adc5bbd17d99fcb5: Status 404 returned error can't find the container with id 7dbb8fbd3842df8f698fc0ba186f62a7fb9bc14a09cfe717adc5bbd17d99fcb5 Nov 02 14:15:48.693773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.693731 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:48.694223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.694019 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:48.694740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.694723 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:15:48.784078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:48.784043 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" event={"ID":"3e862b91-b6c8-44b6-b8d3-ca45d890af9d","Type":"ContainerStarted","Data":"7dbb8fbd3842df8f698fc0ba186f62a7fb9bc14a09cfe717adc5bbd17d99fcb5"} Nov 02 14:15:49.792699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:49.792652 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" event={"ID":"3e862b91-b6c8-44b6-b8d3-ca45d890af9d","Type":"ContainerStarted","Data":"f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a"} Nov 02 14:15:49.793130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:49.792812 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:49.843778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:49.843740 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:49.868135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:49.865948 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" podStartSLOduration=2.489124575 podStartE2EDuration="2.865923809s" totalImagesPullingTime="376.799234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:49.860521886 +0000 UTC m=+1898.586143666" watchObservedRunningTime="2026-11-02 14:15:49.865923809 +0000 UTC m=+1898.591545588" Nov 02 14:15:50.607277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:50.607250 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:15:51.563009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:51.562095 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:53.000178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:52.999943 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:15:53.208977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:53.208946 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:55.070087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:55.070047 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:55.803777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:55.803747 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:15:56.856959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:56.856920 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:58.617977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:58.617936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:15:58.868064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:15:58.867964 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:00.424002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:00.415759 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:01.075476 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:16:01.075430 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:16:02.026629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:02.026585 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:02.188288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:02.188248 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:02.442208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:02.442177 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:02.677649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:02.677588 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:03.896027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:03.889696 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:05.628172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:05.628133 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:07.364656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:07.364617 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:08.968011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:08.966563 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:10.474337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.474304 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb"] Nov 02 14:16:10.478660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.478634 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.513396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.513362 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4c2811\"/\"kube-root-ca.crt\"" Nov 02 14:16:10.513579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.513499 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4c2811\"/\"openshift-service-ca.crt\"" Nov 02 14:16:10.513739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.513718 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4c2811\"/\"auth-test-model-4c2811-kserve-self-signed-certs\"" Nov 02 14:16:10.516587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.516563 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4c2811\"/\"default-dockercfg-ww7fw\"" Nov 02 14:16:10.520067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.520043 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb"] Nov 02 14:16:10.558818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.558782 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-dshm\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.558996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.558946 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tls-certs\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.559062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.559004 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-model-cache\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.559204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.559182 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9q4m\" (UniqueName: \"kubernetes.io/projected/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-kube-api-access-g9q4m\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.559262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.559233 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-home\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.559478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.559460 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tmp-dir\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.595306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.595268 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:10.660521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.660488 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tls-certs\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.660718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.660538 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-model-cache\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.660718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.660636 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9q4m\" (UniqueName: \"kubernetes.io/projected/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-kube-api-access-g9q4m\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.660718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.660675 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-home\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.660912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.660798 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tmp-dir\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.660912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.660878 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-dshm\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.661146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.661117 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-model-cache\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.661271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.661183 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-home\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.661434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.661413 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tmp-dir\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.663297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.663264 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-dshm\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.663515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.663500 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tls-certs\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.686536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.686509 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9q4m\" (UniqueName: \"kubernetes.io/projected/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-kube-api-access-g9q4m\") pod \"auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.801074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.800559 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:10.827517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:10.827472 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:11.227464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:11.227429 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb"] Nov 02 14:16:11.228291 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:16:11.228264 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4abf5d52_3331_44ea_a5b1_71509bb3fc5c.slice/crio-11259a1d605a195ee5cc6d7eb258ce15e4e038fcf1b408310dc642f2b764071b WatchSource:0}: Error finding container 11259a1d605a195ee5cc6d7eb258ce15e4e038fcf1b408310dc642f2b764071b: Status 404 returned error can't find the container with id 11259a1d605a195ee5cc6d7eb258ce15e4e038fcf1b408310dc642f2b764071b Nov 02 14:16:12.021161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:12.021122 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" event={"ID":"4abf5d52-3331-44ea-a5b1-71509bb3fc5c","Type":"ContainerStarted","Data":"40890d5d9d297a45071b00b7ec98e42dd5c384181effddf9c6e817bee102b321"} Nov 02 14:16:12.021161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:12.021162 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" event={"ID":"4abf5d52-3331-44ea-a5b1-71509bb3fc5c","Type":"ContainerStarted","Data":"11259a1d605a195ee5cc6d7eb258ce15e4e038fcf1b408310dc642f2b764071b"} Nov 02 14:16:12.065664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:12.065606 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" podStartSLOduration=2.06559259 podStartE2EDuration="2.06559259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:12.063923978 +0000 UTC m=+1920.789545749" watchObservedRunningTime="2026-11-02 14:16:12.06559259 +0000 UTC m=+1920.791214368" Nov 02 14:16:12.580005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:12.578020 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:12.801578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:12.801539 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:14.239400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:14.239360 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:15.880644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:15.880606 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:16.161265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:16.161230 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z"] Nov 02 14:16:16.161444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:16.161428 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" containerID="cri-o://5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926" gracePeriod=30 Nov 02 14:16:17.665083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:17.665047 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:18.658009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:18.656657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:19.447512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:19.447462 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:20.802129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:20.802092 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:20.814942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:20.814914 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:21.101131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.101048 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:16:21.285604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.285566 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:21.502530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.502503 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 14:16:21.591575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.591535 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-credential-socket\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.591769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.591657 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-socket\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.591843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.591764 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-token\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-token\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.591906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.591863 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-credential-socket" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:21.592028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.591970 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-socket" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:21.592212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.591832 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-kube-api-access-q8lwq\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-kube-api-access-q8lwq\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.592300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.592284 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-podinfo\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.592366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.592324 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-envoy\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.592366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.592362 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79aa1db-ad41-4b56-87a5-74858aa385ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79aa1db-ad41-4b56-87a5-74858aa385ce-istiod-ca-cert\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.592481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.592446 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-certs\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.592546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.592514 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-data\") pod \"b79aa1db-ad41-4b56-87a5-74858aa385ce\" (UID: \"b79aa1db-ad41-4b56-87a5-74858aa385ce\") " Nov 02 14:16:21.593121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.593079 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-certs" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:21.593223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.593161 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-data" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:21.593573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.593498 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79aa1db-ad41-4b56-87a5-74858aa385ce-istiod-ca-cert" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:21.593573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.593503 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:21.593573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.593536 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:21.593573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.593548 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:21.593573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.593557 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:21.594549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.594521 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-token" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:21.595438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.595403 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-podinfo" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:16:21.595438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.595421 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-envoy" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:21.596102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.596081 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-kube-api-access-q8lwq" pod "b79aa1db-ad41-4b56-87a5-74858aa385ce" (UID: "b79aa1db-ad41-4b56-87a5-74858aa385ce"). InnerVolumeSpecName "kube-api-access-q8lwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:21.694892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.694806 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:21.694892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.694836 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8lwq\" (UniqueName: \"kubernetes.io/projected/b79aa1db-ad41-4b56-87a5-74858aa385ce-kube-api-access-q8lwq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:21.694892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.694847 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:21.694892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.694856 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79aa1db-ad41-4b56-87a5-74858aa385ce-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:21.694892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:21.694867 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79aa1db-ad41-4b56-87a5-74858aa385ce-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:16:22.097418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.097331 5591 generic.go:356] "Generic (PLEG): container finished" podID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerID="5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926" exitCode=0 Nov 02 14:16:22.097418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.097378 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" Nov 02 14:16:22.097903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.097415 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" event={"ID":"b79aa1db-ad41-4b56-87a5-74858aa385ce","Type":"ContainerDied","Data":"5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926"} Nov 02 14:16:22.097903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.097475 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z" event={"ID":"b79aa1db-ad41-4b56-87a5-74858aa385ce","Type":"ContainerDied","Data":"fbd00c118efb3d591875f66915282e5413e8ee2856d22c47bda02c9dd232e178"} Nov 02 14:16:22.097903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.097505 5591 scope.go:122] "RemoveContainer" containerID="5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926" Nov 02 14:16:22.108484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.108461 5591 scope.go:122] "RemoveContainer" containerID="5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926" Nov 02 14:16:22.108741 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:16:22.108720 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926\": container with ID starting with 5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926 not found: ID does not exist" containerID="5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926" Nov 02 14:16:22.108807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.108753 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926"} err="failed to get container status \"5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926\": rpc error: code = NotFound desc = could not find container \"5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926\": container with ID starting with 5345d279aaf42906a8661d264c64973990c5b2273cdda61bf2a77f705f66d926 not found: ID does not exist" Nov 02 14:16:22.170023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.169978 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z"] Nov 02 14:16:22.182869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.182840 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-4s85z"] Nov 02 14:16:23.000021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:22.996753 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:23.431623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:23.431590 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" path="/var/lib/kubelet/pods/b79aa1db-ad41-4b56-87a5-74858aa385ce/volumes" Nov 02 14:16:23.672079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:23.672046 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:24.145447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.145415 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2"] Nov 02 14:16:24.146160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.146141 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" Nov 02 14:16:24.146229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.146161 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" Nov 02 14:16:24.146331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.146321 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79aa1db-ad41-4b56-87a5-74858aa385ce" containerName="istio-proxy" Nov 02 14:16:24.151492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.151470 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.172461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.168363 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4b7759\"/\"openshift-service-ca.crt\"" Nov 02 14:16:24.180403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.180331 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4b7759\"/\"kube-root-ca.crt\"" Nov 02 14:16:24.180578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.180469 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4b7759\"/\"default-dockercfg-2s5pr\"" Nov 02 14:16:24.180578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.180552 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4b7759\"/\"auth-test-model-4b7759-kserve-self-signed-certs\"" Nov 02 14:16:24.193297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.193262 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2"] Nov 02 14:16:24.231457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.231422 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrqct\" (UniqueName: \"kubernetes.io/projected/8295ec02-0edd-401e-8973-44ea94e52e90-kube-api-access-jrqct\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.231619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.231485 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-dshm\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.231619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.231574 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-tmp-dir\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.231619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.231612 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-model-cache\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.231796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.231642 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8295ec02-0edd-401e-8973-44ea94e52e90-tls-certs\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.231924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.231899 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-home\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.268339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.268308 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:24.333812 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.333777 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrqct\" (UniqueName: \"kubernetes.io/projected/8295ec02-0edd-401e-8973-44ea94e52e90-kube-api-access-jrqct\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.334024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.333824 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-dshm\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.334024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.333873 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-tmp-dir\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.334024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.333906 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-model-cache\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.334024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.333933 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8295ec02-0edd-401e-8973-44ea94e52e90-tls-certs\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.334296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.334134 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-home\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.334458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.334433 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-tmp-dir\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.334537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.334457 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-model-cache\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.334537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.334502 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-home\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.336368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.336343 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-dshm\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.336615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.336598 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8295ec02-0edd-401e-8973-44ea94e52e90-tls-certs\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.358234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.358202 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrqct\" (UniqueName: \"kubernetes.io/projected/8295ec02-0edd-401e-8973-44ea94e52e90-kube-api-access-jrqct\") pod \"auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.476062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.467544 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:24.746034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.745832 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:24.875648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:24.875615 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2"] Nov 02 14:16:24.876751 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:16:24.876721 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8295ec02_0edd_401e_8973_44ea94e52e90.slice/crio-d6551022585e50ca959d1bfed363ac5d74a27996a46134931e583c032ceeb782 WatchSource:0}: Error finding container d6551022585e50ca959d1bfed363ac5d74a27996a46134931e583c032ceeb782: Status 404 returned error can't find the container with id d6551022585e50ca959d1bfed363ac5d74a27996a46134931e583c032ceeb782 Nov 02 14:16:25.121444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:25.121356 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" event={"ID":"8295ec02-0edd-401e-8973-44ea94e52e90","Type":"ContainerStarted","Data":"c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e"} Nov 02 14:16:25.121444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:25.121405 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" event={"ID":"8295ec02-0edd-401e-8973-44ea94e52e90","Type":"ContainerStarted","Data":"d6551022585e50ca959d1bfed363ac5d74a27996a46134931e583c032ceeb782"} Nov 02 14:16:25.165490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:25.165433 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" podStartSLOduration=1.165418101 podStartE2EDuration="1.165418101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:25.163446223 +0000 UTC m=+1933.889067992" watchObservedRunningTime="2026-11-02 14:16:25.165418101 +0000 UTC m=+1933.891039870" Nov 02 14:16:25.468663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:25.468626 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:26.387009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:26.377832 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:27.957141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:27.957106 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:16:28.016013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:28.013417 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:30.008349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:30.008031 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:31.075132 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:16:31.075095 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:16:31.868036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:31.867997 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:32.062919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:32.062887 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:32.381532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:32.381486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:32.550423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:32.550388 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:33.584199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:33.584162 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:34.467922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:34.467889 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:34.480662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:34.480636 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:35.299006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:35.298953 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:16:35.356105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:35.356052 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:36.783459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:36.783399 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:37.097599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:37.097509 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:38.958007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:38.955998 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:40.721625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:40.721585 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:42.594136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:42.594098 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:44.286923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:44.286876 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:45.849579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:45.849538 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:47.467601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:47.467556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:49.284011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:49.280826 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:51.148006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:51.147957 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:52.784584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:52.784521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:54.570057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:54.570005 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:56.167008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:56.166358 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:57.069141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.069104 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59"] Nov 02 14:16:57.076265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.076237 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.099940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.099907 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59"] Nov 02 14:16:57.102836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.102810 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5cbbfb\"/\"kube-root-ca.crt\"" Nov 02 14:16:57.107499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.107473 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5cbbfb\"/\"openshift-service-ca.crt\"" Nov 02 14:16:57.115919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.115884 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5cbbfb\"/\"default-dockercfg-bdjf7\"" Nov 02 14:16:57.116079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.115956 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5cbbfb\"/\"explicit-model-5cbbfb-kserve-self-signed-certs\"" Nov 02 14:16:57.192917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.192879 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-model-cache\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.193340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.192947 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-home\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.193340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.193060 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tmp-dir\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.193340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.193265 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tls-certs\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.193340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.193292 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-dshm\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.193340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.193326 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7966\" (UniqueName: \"kubernetes.io/projected/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-kube-api-access-w7966\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.224126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.224064 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:57.294361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294321 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-model-cache\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.294551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294418 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-home\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.294551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294460 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tmp-dir\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.294695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294677 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tls-certs\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.294757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294711 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-dshm\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.294825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294762 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7966\" (UniqueName: \"kubernetes.io/projected/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-kube-api-access-w7966\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.294825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294782 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-model-cache\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.294921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294844 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-home\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.294921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.294860 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tmp-dir\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.297086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.297060 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-dshm\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.297346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.297317 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tls-certs\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.321878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.321817 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7966\" (UniqueName: \"kubernetes.io/projected/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-kube-api-access-w7966\") pod \"explicit-model-5cbbfb-kserve-588fc4779-jft59\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.449300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.449260 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:57.943833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:57.943800 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59"] Nov 02 14:16:57.945774 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:16:57.945746 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2c1b741_6c1e_4e8c_bdb9_8597598366a0.slice/crio-b148cb3c23544d5af144e8017810d479c6be4b538c5ce8306d9cc2f502f032a6 WatchSource:0}: Error finding container b148cb3c23544d5af144e8017810d479c6be4b538c5ce8306d9cc2f502f032a6: Status 404 returned error can't find the container with id b148cb3c23544d5af144e8017810d479c6be4b538c5ce8306d9cc2f502f032a6 Nov 02 14:16:58.137516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:58.137471 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:16:58.635260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:58.634975 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" event={"ID":"e2c1b741-6c1e-4e8c-bdb9-8597598366a0","Type":"ContainerStarted","Data":"62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93"} Nov 02 14:16:58.635260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:58.635036 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" event={"ID":"e2c1b741-6c1e-4e8c-bdb9-8597598366a0","Type":"ContainerStarted","Data":"b148cb3c23544d5af144e8017810d479c6be4b538c5ce8306d9cc2f502f032a6"} Nov 02 14:16:58.683882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:58.683826 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" podStartSLOduration=1.683813118 podStartE2EDuration="1.683813118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:58.681025987 +0000 UTC m=+1967.406647752" watchObservedRunningTime="2026-11-02 14:16:58.683813118 +0000 UTC m=+1967.409434887" Nov 02 14:16:59.463554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:59.463525 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:16:59.944525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:16:59.944486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:01.074070 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:17:01.074021 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:17:01.602476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:01.602427 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:03.311106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:03.310864 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:04.952015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:04.948616 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:06.666010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:06.664063 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:07.463920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:07.463880 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:17:07.476862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:07.476831 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:17:07.602610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:07.602578 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:07.895965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:07.895936 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:17:08.298550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:08.288746 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:10.116279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:10.116151 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:11.123767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:11.123733 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59"] Nov 02 14:17:11.239071 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:11.238942 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:11.695924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:11.695862 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:11.896954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:11.895418 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:12.758173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.758131 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4"] Nov 02 14:17:12.764953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.764930 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:12.782887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.782855 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4"] Nov 02 14:17:12.791929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.791899 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-0d1ca7\"/\"kube-root-ca.crt\"" Nov 02 14:17:12.791929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.791900 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-0d1ca7\"/\"openshift-service-ca.crt\"" Nov 02 14:17:12.794157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.794135 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-0d1ca7\"/\"default-dockercfg-lbn2c\"" Nov 02 14:17:12.794285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.794178 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-0d1ca7\"/\"auth-test-model-0d1ca7-kserve-self-signed-certs\"" Nov 02 14:17:12.878047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.877952 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:12.930568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.930480 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-dshm\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:12.930731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.930666 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tls-certs\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:12.930731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.930721 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96j8x\" (UniqueName: \"kubernetes.io/projected/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-kube-api-access-96j8x\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:12.930820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.930799 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tmp-dir\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:12.930920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.930902 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-model-cache\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:12.931001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:12.930970 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-home\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.047329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.047273 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-model-cache\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.049713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.049683 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-home\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.050390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.050365 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-dshm\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.050732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.050708 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tls-certs\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.050919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.050904 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96j8x\" (UniqueName: \"kubernetes.io/projected/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-kube-api-access-96j8x\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.051098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.051080 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tmp-dir\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.051607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.051585 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tmp-dir\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.051750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.049547 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-model-cache\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.051859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.050252 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-home\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.075025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.074972 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tls-certs\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.107841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.107802 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96j8x\" (UniqueName: \"kubernetes.io/projected/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-kube-api-access-96j8x\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.116383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.116348 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-dshm\") pod \"auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.384609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.384574 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:13.689073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.685582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:13.745339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.745309 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4"] Nov 02 14:17:13.746052 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:17:13.746019 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03bf3b4b_34d9_4c38_a24b_3f00f22c1e33.slice/crio-533efda8142113022457d67e2163ed34ddf7bf6889a330c12d17c789a9ab23c9 WatchSource:0}: Error finding container 533efda8142113022457d67e2163ed34ddf7bf6889a330c12d17c789a9ab23c9: Status 404 returned error can't find the container with id 533efda8142113022457d67e2163ed34ddf7bf6889a330c12d17c789a9ab23c9 Nov 02 14:17:13.813264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.813236 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" event={"ID":"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33","Type":"ContainerStarted","Data":"533efda8142113022457d67e2163ed34ddf7bf6889a330c12d17c789a9ab23c9"} Nov 02 14:17:13.859019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.858948 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" podStartSLOduration=1.858933266 podStartE2EDuration="1.858933266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:13.856730302 +0000 UTC m=+1982.582352073" watchObservedRunningTime="2026-11-02 14:17:13.858933266 +0000 UTC m=+1982.584555036" Nov 02 14:17:13.953316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:13.953232 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:17:14.385712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:14.385675 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:14.853064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:14.847625 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" event={"ID":"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33","Type":"ContainerStarted","Data":"fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031"} Nov 02 14:17:15.565206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:15.565168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:16.722205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:16.722160 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:17.274809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:17.274765 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:18.948515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:18.948477 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:20.581143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:20.581054 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:22.384009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:22.379933 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:23.385050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:23.385010 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:23.399417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:23.399391 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:23.792321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:23.792240 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:23.919786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:23.919751 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:17:24.044338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:24.044205 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:25.763625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:25.763567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:26.145409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.145371 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" podUID="e2c1b741-6c1e-4e8c-bdb9-8597598366a0" containerName="main" containerID="cri-o://62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93" gracePeriod=45 Nov 02 14:17:26.432561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.432533 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:17:26.603312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603272 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tmp-dir\") pod \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " Nov 02 14:17:26.603488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603335 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-model-cache\") pod \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " Nov 02 14:17:26.603488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603439 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-kube-api-access-w7966\" (UniqueName: \"kubernetes.io/projected/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-kube-api-access-w7966\") pod \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " Nov 02 14:17:26.603605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603583 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-home\") pod \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " Nov 02 14:17:26.603661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603599 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tmp-dir" pod "e2c1b741-6c1e-4e8c-bdb9-8597598366a0" (UID: "e2c1b741-6c1e-4e8c-bdb9-8597598366a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:26.603661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603611 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-model-cache" pod "e2c1b741-6c1e-4e8c-bdb9-8597598366a0" (UID: "e2c1b741-6c1e-4e8c-bdb9-8597598366a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:26.603802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603782 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tls-certs\") pod \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " Nov 02 14:17:26.603868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603805 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-home" pod "e2c1b741-6c1e-4e8c-bdb9-8597598366a0" (UID: "e2c1b741-6c1e-4e8c-bdb9-8597598366a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:26.603868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.603826 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-dshm\") pod \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\" (UID: \"e2c1b741-6c1e-4e8c-bdb9-8597598366a0\") " Nov 02 14:17:26.604509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.604487 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:17:26.604613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.604511 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:17:26.604613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.604523 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:17:26.606021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.605967 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-kube-api-access-w7966" pod "e2c1b741-6c1e-4e8c-bdb9-8597598366a0" (UID: "e2c1b741-6c1e-4e8c-bdb9-8597598366a0"). InnerVolumeSpecName "kube-api-access-w7966". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:26.606021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.605970 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-dshm" pod "e2c1b741-6c1e-4e8c-bdb9-8597598366a0" (UID: "e2c1b741-6c1e-4e8c-bdb9-8597598366a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:26.606203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.606036 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tls-certs" pod "e2c1b741-6c1e-4e8c-bdb9-8597598366a0" (UID: "e2c1b741-6c1e-4e8c-bdb9-8597598366a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:17:26.683278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.683200 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:26.705974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.705938 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:17:26.705974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.705968 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:17:26.705974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.705977 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7966\" (UniqueName: \"kubernetes.io/projected/e2c1b741-6c1e-4e8c-bdb9-8597598366a0-kube-api-access-w7966\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:17:26.971878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.971787 5591 generic.go:356] "Generic (PLEG): container finished" podID="e2c1b741-6c1e-4e8c-bdb9-8597598366a0" containerID="62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93" exitCode=0 Nov 02 14:17:26.972355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.972103 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" Nov 02 14:17:26.972355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.972109 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" event={"ID":"e2c1b741-6c1e-4e8c-bdb9-8597598366a0","Type":"ContainerDied","Data":"62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93"} Nov 02 14:17:26.972355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.972158 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59" event={"ID":"e2c1b741-6c1e-4e8c-bdb9-8597598366a0","Type":"ContainerDied","Data":"b148cb3c23544d5af144e8017810d479c6be4b538c5ce8306d9cc2f502f032a6"} Nov 02 14:17:26.972355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.972189 5591 scope.go:122] "RemoveContainer" containerID="62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93" Nov 02 14:17:26.988255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.988232 5591 scope.go:122] "RemoveContainer" containerID="62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93" Nov 02 14:17:26.988601 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:17:26.988574 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93\": container with ID starting with 62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93 not found: ID does not exist" containerID="62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93" Nov 02 14:17:26.988677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:26.988614 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93"} err="failed to get container status \"62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93\": rpc error: code = NotFound desc = could not find container \"62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93\": container with ID starting with 62dcbb9527b8d83e5e81c817c0287046837a5c012ddaf2af5ed67452dc1fae93 not found: ID does not exist" Nov 02 14:17:27.027031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:27.026999 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59"] Nov 02 14:17:27.039784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:27.039760 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5cbbfb/explicit-model-5cbbfb-kserve-588fc4779-jft59"] Nov 02 14:17:27.445355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:27.445313 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:27.451370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:27.451188 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c1b741-6c1e-4e8c-bdb9-8597598366a0" path="/var/lib/kubelet/pods/e2c1b741-6c1e-4e8c-bdb9-8597598366a0/volumes" Nov 02 14:17:29.059977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:29.059917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:30.675047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:30.674651 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:31.074662 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:17:31.074566 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:17:32.562710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:32.562658 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:34.257006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:34.256730 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:35.950565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:35.950527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:37.676010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:37.675836 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:39.264765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:39.264700 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:40.879008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:40.874919 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:42.210950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:42.210916 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:42.554596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:42.554431 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:44.140552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:44.140493 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:45.778080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:45.778037 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:47.458334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:47.458294 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:49.142525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:49.142486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:50.800388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:50.800326 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:52.559116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:52.559079 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:53.343315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:53.343276 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn"] Nov 02 14:17:53.508377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:53.508340 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:53.699666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:53.699630 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:54.591844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:54.586817 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:54.779722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:54.779689 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:56.087178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.087143 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2"] Nov 02 14:17:56.087792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.087776 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e2c1b741-6c1e-4e8c-bdb9-8597598366a0" containerName="main" Nov 02 14:17:56.087839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.087794 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c1b741-6c1e-4e8c-bdb9-8597598366a0" containerName="main" Nov 02 14:17:56.087940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.087930 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c1b741-6c1e-4e8c-bdb9-8597598366a0" containerName="main" Nov 02 14:17:56.091904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.091883 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.112260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.112239 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-2cddf2-openshift-default-dockercfg-8zz64\"" Nov 02 14:17:56.127728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.127702 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2"] Nov 02 14:17:56.214102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214064 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.214102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214102 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9wt\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-kube-api-access-gx9wt\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.214384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214284 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.214443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214420 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.214498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214484 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.214542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214518 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.214587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214544 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.214628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214612 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03005eb-15b1-4d0b-a99b-81e0f207be63-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.214663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.214635 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316237 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316294 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03005eb-15b1-4d0b-a99b-81e0f207be63-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316333 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316454 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316479 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9wt\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-kube-api-access-gx9wt\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316603 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316715 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316766 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.316895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.316822 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.317107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.317050 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.317107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.317091 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.317304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.317129 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.317304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.317151 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.317449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.317297 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03005eb-15b1-4d0b-a99b-81e0f207be63-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.318799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.318774 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.319137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.319120 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.341786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.341723 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.341907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.341883 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9wt\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-kube-api-access-gx9wt\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.406610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.406379 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:56.407144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.407115 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:56.799890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.799850 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2"] Nov 02 14:17:56.803234 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:17:56.803199 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03005eb_15b1_4d0b_a99b_81e0f207be63.slice/crio-e56102d39beea8915020ead9385276f0f2bf1a58062c2f27d055620dc2b1483c WatchSource:0}: Error finding container e56102d39beea8915020ead9385276f0f2bf1a58062c2f27d055620dc2b1483c: Status 404 returned error can't find the container with id e56102d39beea8915020ead9385276f0f2bf1a58062c2f27d055620dc2b1483c Nov 02 14:17:56.807198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.807163 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:17:56.807338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.807247 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:17:56.807338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:56.807292 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:17:57.272170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:57.272130 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" event={"ID":"a03005eb-15b1-4d0b-a99b-81e0f207be63","Type":"ContainerStarted","Data":"cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290"} Nov 02 14:17:57.272574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:57.272189 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" event={"ID":"a03005eb-15b1-4d0b-a99b-81e0f207be63","Type":"ContainerStarted","Data":"e56102d39beea8915020ead9385276f0f2bf1a58062c2f27d055620dc2b1483c"} Nov 02 14:17:57.330479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:57.330416 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podStartSLOduration=1.330398934 podStartE2EDuration="1.330398934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:57.327213373 +0000 UTC m=+2026.052835145" watchObservedRunningTime="2026-11-02 14:17:57.330398934 +0000 UTC m=+2026.056020705" Nov 02 14:17:57.408283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:57.408251 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:17:57.410121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:57.410088 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 02 14:17:57.410246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:57.410159 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 02 14:17:58.171582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.171547 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:17:58.349016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.348407 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" podUID="37d3b616-82de-42fd-9990-97fef65c2781" containerName="payload-pre-processing" containerID="cri-o://78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862" gracePeriod=25 Nov 02 14:17:58.349958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.349137 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:17:58.408939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.408910 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 02 14:17:58.409105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.408961 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 02 14:17:58.698711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.698682 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:17:58.780290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.780197 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d3b616-82de-42fd-9990-97fef65c2781-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d3b616-82de-42fd-9990-97fef65c2781-plugins-config-volume\") pod \"37d3b616-82de-42fd-9990-97fef65c2781\" (UID: \"37d3b616-82de-42fd-9990-97fef65c2781\") " Nov 02 14:17:58.780463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.780401 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d3b616-82de-42fd-9990-97fef65c2781-kube-api-access-fvbdl\" (UniqueName: \"kubernetes.io/projected/37d3b616-82de-42fd-9990-97fef65c2781-kube-api-access-fvbdl\") pod \"37d3b616-82de-42fd-9990-97fef65c2781\" (UID: \"37d3b616-82de-42fd-9990-97fef65c2781\") " Nov 02 14:17:58.780706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.780677 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d3b616-82de-42fd-9990-97fef65c2781-plugins-config-volume" pod "37d3b616-82de-42fd-9990-97fef65c2781" (UID: "37d3b616-82de-42fd-9990-97fef65c2781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:58.781179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.781150 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d3b616-82de-42fd-9990-97fef65c2781-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:17:58.782681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.782648 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d3b616-82de-42fd-9990-97fef65c2781-kube-api-access-fvbdl" pod "37d3b616-82de-42fd-9990-97fef65c2781" (UID: "37d3b616-82de-42fd-9990-97fef65c2781"). InnerVolumeSpecName "kube-api-access-fvbdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:58.882619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:58.882581 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvbdl\" (UniqueName: \"kubernetes.io/projected/37d3b616-82de-42fd-9990-97fef65c2781-kube-api-access-fvbdl\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:17:59.331015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.323429 5591 generic.go:356] "Generic (PLEG): container finished" podID="37d3b616-82de-42fd-9990-97fef65c2781" containerID="78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862" exitCode=0 Nov 02 14:17:59.331015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.324685 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" Nov 02 14:17:59.331015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.325459 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" event={"ID":"37d3b616-82de-42fd-9990-97fef65c2781","Type":"ContainerDied","Data":"78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862"} Nov 02 14:17:59.331015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.325513 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn" event={"ID":"37d3b616-82de-42fd-9990-97fef65c2781","Type":"ContainerDied","Data":"22661ff7aef234ec6edc01ad4059f2cbd89f0104a912588b185298907d3457f8"} Nov 02 14:17:59.331015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.325547 5591 scope.go:122] "RemoveContainer" containerID="78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862" Nov 02 14:17:59.375634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.368630 5591 scope.go:122] "RemoveContainer" containerID="78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862" Nov 02 14:17:59.376000 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:17:59.375679 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862\": container with ID starting with 78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862 not found: ID does not exist" containerID="78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862" Nov 02 14:17:59.376000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.375724 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862"} err="failed to get container status \"78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862\": rpc error: code = NotFound desc = could not find container \"78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862\": container with ID starting with 78ad19b0f911c3bbca6f88104e75fcb1d1a852e852abfae8c436f094ed226862 not found: ID does not exist" Nov 02 14:17:59.411019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.408460 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 02 14:17:59.411019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.408509 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 02 14:17:59.411292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.411219 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn"] Nov 02 14:17:59.423500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.423465 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5cbbfb-58dcc8968c-gnsbn"] Nov 02 14:17:59.452035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:17:59.444010 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d3b616-82de-42fd-9990-97fef65c2781" path="/var/lib/kubelet/pods/37d3b616-82de-42fd-9990-97fef65c2781/volumes" Nov 02 14:18:00.244009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:00.229335 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:00.408499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:00.408463 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 02 14:18:00.408906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:00.408516 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 02 14:18:01.081834 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:18:01.081797 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:18:01.408246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:01.408206 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 02 14:18:01.408442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:01.408266 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 02 14:18:02.090336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:02.090302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:02.260029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:02.258729 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:02.408957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:02.408924 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 02 14:18:02.409181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:02.408978 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 02 14:18:03.408663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:03.408623 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 02 14:18:03.409090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:03.408690 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 02 14:18:03.829351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:03.829263 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:04.413463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:04.413435 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:18:04.413934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:04.413915 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:18:04.414670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:04.414646 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:18:05.407315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:05.407258 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:07.179815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:07.179774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:08.856097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:08.855939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:10.449060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:10.449017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:11.924186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:11.924147 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:12.734420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:12.734379 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:14.430536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:14.430494 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:16.065052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:16.065010 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:18.070008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:18.066072 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:18.666918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:18.666861 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:18.835078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:18.835041 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:19.490819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:19.490779 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:19.855029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:19.854860 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:20.026595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:20.026554 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:20.610211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:20.610180 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:21.163006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:21.159943 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:21.511579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:21.511462 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:21.778052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:21.777429 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:23.150649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:23.150612 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:25.105962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:25.105928 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:25.360548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:25.360462 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:26.281338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:26.281304 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:27.104008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:27.098978 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:28.765193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:28.765094 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:30.370498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:30.370457 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:31.074388 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:18:31.074343 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:18:32.044022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:32.043965 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:33.657462 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:33.657415 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:34.657579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.657545 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k"] Nov 02 14:18:34.658233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.658216 5591 cpu_manager.go:414] "containerMap: removing container" podUID="37d3b616-82de-42fd-9990-97fef65c2781" containerName="payload-pre-processing" Nov 02 14:18:34.658286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.658234 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d3b616-82de-42fd-9990-97fef65c2781" containerName="payload-pre-processing" Nov 02 14:18:34.658370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.658361 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d3b616-82de-42fd-9990-97fef65c2781" containerName="payload-pre-processing" Nov 02 14:18:34.663491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.663467 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.680040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.679971 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-2daae4-openshift-default-dockercfg-tf6d9\"" Nov 02 14:18:34.701009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.700953 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k"] Nov 02 14:18:34.828531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.828495 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-credential-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.828531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.828544 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-data\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.828797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.828666 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.828869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.828803 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-token\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.828961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.828944 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-certs\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.829086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.829070 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4258b1b8-9638-4a7f-973c-8209d717bfd0-istiod-ca-cert\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.829155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.829140 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-podinfo\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.829217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.829207 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-envoy\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.829338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.829280 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfn8m\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-kube-api-access-xfn8m\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930317 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-certs\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930383 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4258b1b8-9638-4a7f-973c-8209d717bfd0-istiod-ca-cert\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930415 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-podinfo\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930445 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-envoy\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930504 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfn8m\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-kube-api-access-xfn8m\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930578 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-credential-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930617 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-data\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930697 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930803 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-token\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.930958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.930815 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-certs\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.931187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.931030 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-credential-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.931187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.931162 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-data\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.931319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.931276 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4258b1b8-9638-4a7f-973c-8209d717bfd0-istiod-ca-cert\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.931371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.931341 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.932822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.932795 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-envoy\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.933086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.933066 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-podinfo\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.956251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.956228 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfn8m\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-kube-api-access-xfn8m\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.957315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.957296 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-token\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:34.978950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:34.978917 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:35.298075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.291093 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:35.331072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.326411 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k"] Nov 02 14:18:35.338184 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:18:35.338147 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4258b1b8_9638_4a7f_973c_8209d717bfd0.slice/crio-c84d44fc38f0511fecdcf3c672efa76aa1fc18ab089e5a25651ef1c85f8f98ab WatchSource:0}: Error finding container c84d44fc38f0511fecdcf3c672efa76aa1fc18ab089e5a25651ef1c85f8f98ab: Status 404 returned error can't find the container with id c84d44fc38f0511fecdcf3c672efa76aa1fc18ab089e5a25651ef1c85f8f98ab Nov 02 14:18:35.340716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.340678 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:18:35.340818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.340754 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:18:35.340818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.340783 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:18:35.678896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.678855 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" event={"ID":"4258b1b8-9638-4a7f-973c-8209d717bfd0","Type":"ContainerStarted","Data":"76346f0d9af6d0393e0396a952dfd6b724bfc4c8052f949f33b688e193411a4c"} Nov 02 14:18:35.679496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.678915 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" event={"ID":"4258b1b8-9638-4a7f-973c-8209d717bfd0","Type":"ContainerStarted","Data":"c84d44fc38f0511fecdcf3c672efa76aa1fc18ab089e5a25651ef1c85f8f98ab"} Nov 02 14:18:35.729468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.729413 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" podStartSLOduration=1.729397918 podStartE2EDuration="1.729397918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:35.726268117 +0000 UTC m=+2064.451889886" watchObservedRunningTime="2026-11-02 14:18:35.729397918 +0000 UTC m=+2064.455019688" Nov 02 14:18:35.979153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:35.979072 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:36.980799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:36.980761 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 14:18:36.981223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:36.980816 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 14:18:37.033073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:37.028942 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:37.757522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:37.757479 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:18:37.979358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:37.979316 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 14:18:37.979541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:37.979380 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 14:18:38.589164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:38.589073 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:38.979901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:38.979859 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 14:18:38.980166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:38.979914 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 14:18:39.980039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:39.979975 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 14:18:39.980434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:39.980057 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 14:18:40.456013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:40.454363 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:40.979363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:40.979326 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 14:18:40.979573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:40.979391 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 14:18:41.984178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:41.984144 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:41.984677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:41.984658 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:41.985374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:41.985354 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:18:42.142080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:42.142045 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:43.771969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:43.770069 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:45.380010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:45.374717 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:47.087343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:47.087294 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:48.783035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:48.780629 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:50.452185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:50.452142 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:51.344018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:51.340966 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:51.534158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:51.533951 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:51.830356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:51.830318 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:51.993374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:51.993343 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:52.341443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:52.341379 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:54.009342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:54.009085 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:55.724654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:55.724615 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:57.006773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:57.006734 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:57.295072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:57.294961 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:18:58.889389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:18:58.886865 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:00.603327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:00.603266 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:01.074491 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:19:01.074453 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:19:02.269689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:02.269628 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:03.945792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:03.945754 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:05.604043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:05.599898 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:07.417310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:07.417247 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:09.062056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:09.061978 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:10.716205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:10.716166 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:11.918862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:11.918834 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:19:11.944368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:11.944342 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:19:12.591410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:12.591371 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:14.179008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:14.176754 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:15.861571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:15.861532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:17.528191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:17.528149 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:19.071034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:19.070611 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:20.714757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:20.714715 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:22.440011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:22.437322 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:24.096788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:24.096751 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:24.269511 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:24.269468 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:25.189000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:25.188951 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:19:25.775697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:25.775644 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:27.647789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:27.647750 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:29.361953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:29.361917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:30.945663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:30.945620 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:31.077121 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:19:31.077079 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:19:31.321528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:31.321414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:32.662006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:32.661278 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:33.410016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:33.409380 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:34.477007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:34.475973 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:34.769940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:34.769849 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:35.304721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:35.304680 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn"] Nov 02 14:19:35.305014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:35.304965 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" containerID="cri-o://16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158" gracePeriod=30 Nov 02 14:19:36.182748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:36.182061 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:37.867894 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:37.867797 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:38.061934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:38.061892 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:38.735112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:38.735066 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:38.975908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:38.975794 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:39.229373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:39.229335 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7cjzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:19:39.229542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:39.229406 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:19:40.006642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.006584 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:40.590883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.590663 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 14:19:40.663273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663237 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istiod-ca-cert\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.663532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663409 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-socket\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.663532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663444 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-credential-socket\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.663743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663547 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-token\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-token\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.663743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663577 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-podinfo\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.663743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663622 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-kube-api-access-g2cgq\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-kube-api-access-g2cgq\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.663743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663668 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-certs\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.664066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663763 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-envoy\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.664066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.663813 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-data\") pod \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\" (UID: \"b9def5c6-d319-49bd-9a73-3dfdfcd14843\") " Nov 02 14:19:40.664860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.664810 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-data" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:40.665154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.665125 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-socket" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:40.665975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.665918 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-certs" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:40.666298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.666262 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istiod-ca-cert" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:40.666671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.666649 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-credential-socket" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:40.667413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.667387 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-podinfo" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:19:40.671500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.671361 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-envoy" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:40.671500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.671464 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-token" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:40.673156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.673129 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-kube-api-access-g2cgq" pod "b9def5c6-d319-49bd-9a73-3dfdfcd14843" (UID: "b9def5c6-d319-49bd-9a73-3dfdfcd14843"). InnerVolumeSpecName "kube-api-access-g2cgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780003 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780040 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780054 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2cgq\" (UniqueName: \"kubernetes.io/projected/b9def5c6-d319-49bd-9a73-3dfdfcd14843-kube-api-access-g2cgq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780087 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780101 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780115 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780152 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9def5c6-d319-49bd-9a73-3dfdfcd14843-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780165 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:40.781122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:40.780177 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9def5c6-d319-49bd-9a73-3dfdfcd14843-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:41.271112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.271073 5591 generic.go:356] "Generic (PLEG): container finished" podID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerID="16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158" exitCode=0 Nov 02 14:19:41.271629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.271126 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" Nov 02 14:19:41.271629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.271162 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" event={"ID":"b9def5c6-d319-49bd-9a73-3dfdfcd14843","Type":"ContainerDied","Data":"16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158"} Nov 02 14:19:41.271629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.271222 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn" event={"ID":"b9def5c6-d319-49bd-9a73-3dfdfcd14843","Type":"ContainerDied","Data":"569c6fd765f86242616ed3cbb17d4b50660bf6f7732acd2779b0224891733f89"} Nov 02 14:19:41.271629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.271235 5591 scope.go:122] "RemoveContainer" containerID="16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158" Nov 02 14:19:41.281808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.281793 5591 scope.go:122] "RemoveContainer" containerID="16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158" Nov 02 14:19:41.282106 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:19:41.282084 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158\": container with ID starting with 16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158 not found: ID does not exist" containerID="16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158" Nov 02 14:19:41.282163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.282113 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158"} err="failed to get container status \"16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158\": rpc error: code = NotFound desc = could not find container \"16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158\": container with ID starting with 16aa0babb6af291d734456c8e9ebb327cb4d40b962a6a9bdbdfbe1fba7161158 not found: ID does not exist" Nov 02 14:19:41.340343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.340274 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn"] Nov 02 14:19:41.344146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.344122 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cjzn"] Nov 02 14:19:41.433858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.433814 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" path="/var/lib/kubelet/pods/b9def5c6-d319-49bd-9a73-3dfdfcd14843/volumes" Nov 02 14:19:41.664222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:41.664185 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:43.147747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.147714 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:43.316078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.316038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:43.634602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.634565 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:43.950715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.950631 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s"] Nov 02 14:19:43.950867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.950823 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" containerID="cri-o://f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66" gracePeriod=30 Nov 02 14:19:43.964038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.964004 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5"] Nov 02 14:19:43.964254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.964228 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" containerID="cri-o://45192053bf7cb9a2e18f0afb88a9a46df1b9b0266f37afa24614e5ac5e3a7a15" gracePeriod=30 Nov 02 14:19:43.967861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.967817 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v"] Nov 02 14:19:43.968218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.968079 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" containerID="cri-o://2fd38e9a358b501f9b2356e49db93c0601fc7aeba0681bd5beca34e3bb0352af" gracePeriod=30 Nov 02 14:19:43.979195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.979083 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr"] Nov 02 14:19:43.979373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:43.979347 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" containerID="cri-o://1d9684187d171283d535e2ab15e9f75ad71f2496534120879eb2b8f296694dc6" gracePeriod=30 Nov 02 14:19:44.917692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:44.917643 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:45.163016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:45.161557 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:45.787008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:45.783656 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:47.444010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:47.443950 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:47.801601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:47.801522 5591 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:19:47.801601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:47.801580 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:19:49.090783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.080972 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:49.323472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.323443 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.363767 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-envoy\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.363910 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-certs\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.363945 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-socket\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.364146 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-kube-api-access-fc55x\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-kube-api-access-fc55x\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.364184 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-token\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-token\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.364248 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-podinfo\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.364743 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-credential-socket\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.364794 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-data\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.364930 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2465170c-9935-4a55-a00d-7dcaeeb78338-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2465170c-9935-4a55-a00d-7dcaeeb78338-istiod-ca-cert\") pod \"2465170c-9935-4a55-a00d-7dcaeeb78338\" (UID: \"2465170c-9935-4a55-a00d-7dcaeeb78338\") " Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.365435 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-certs" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.365809 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.365874 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2465170c-9935-4a55-a00d-7dcaeeb78338-istiod-ca-cert" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.366048 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-socket" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.366259 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-credential-socket" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.368276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.366275 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-data" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.369851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.369812 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-token" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:49.372617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.372589 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-envoy" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.373791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.373764 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-podinfo" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:19:49.374037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.374017 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-kube-api-access-fc55x" pod "2465170c-9935-4a55-a00d-7dcaeeb78338" (UID: "2465170c-9935-4a55-a00d-7dcaeeb78338"). InnerVolumeSpecName "kube-api-access-fc55x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:49.418761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.418731 5591 generic.go:356] "Generic (PLEG): container finished" podID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerID="1d9684187d171283d535e2ab15e9f75ad71f2496534120879eb2b8f296694dc6" exitCode=0 Nov 02 14:19:49.418945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.418850 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" event={"ID":"27d9e0b3-69fc-4536-b3e2-be7e8d6714be","Type":"ContainerDied","Data":"1d9684187d171283d535e2ab15e9f75ad71f2496534120879eb2b8f296694dc6"} Nov 02 14:19:49.420595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.420558 5591 generic.go:356] "Generic (PLEG): container finished" podID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerID="f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66" exitCode=0 Nov 02 14:19:49.420743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.420631 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" Nov 02 14:19:49.420743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.420638 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" event={"ID":"2465170c-9935-4a55-a00d-7dcaeeb78338","Type":"ContainerDied","Data":"f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66"} Nov 02 14:19:49.420879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.420770 5591 scope.go:122] "RemoveContainer" containerID="f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66" Nov 02 14:19:49.421279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.421157 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s" event={"ID":"2465170c-9935-4a55-a00d-7dcaeeb78338","Type":"ContainerDied","Data":"d5c9b2e9b75bb3c5d8abd88cb89890a7496938dcc3d1d4eb77f0df8566527eda"} Nov 02 14:19:49.427900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.425325 5591 generic.go:356] "Generic (PLEG): container finished" podID="4b605f12-2143-4e18-8a4d-87945d237404" containerID="2fd38e9a358b501f9b2356e49db93c0601fc7aeba0681bd5beca34e3bb0352af" exitCode=0 Nov 02 14:19:49.427900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.427428 5591 generic.go:356] "Generic (PLEG): container finished" podID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerID="45192053bf7cb9a2e18f0afb88a9a46df1b9b0266f37afa24614e5ac5e3a7a15" exitCode=0 Nov 02 14:19:49.438918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.438889 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" event={"ID":"4b605f12-2143-4e18-8a4d-87945d237404","Type":"ContainerDied","Data":"2fd38e9a358b501f9b2356e49db93c0601fc7aeba0681bd5beca34e3bb0352af"} Nov 02 14:19:49.439021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.438947 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" event={"ID":"f3cfb3f5-5903-47dc-9837-27c0f18adabb","Type":"ContainerDied","Data":"45192053bf7cb9a2e18f0afb88a9a46df1b9b0266f37afa24614e5ac5e3a7a15"} Nov 02 14:19:49.441201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.441181 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:19:49.446809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.446788 5591 scope.go:122] "RemoveContainer" containerID="f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66" Nov 02 14:19:49.447674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.446976 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:19:49.447674 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:19:49.447070 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66\": container with ID starting with f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66 not found: ID does not exist" containerID="f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66" Nov 02 14:19:49.447674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.447102 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66"} err="failed to get container status \"f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66\": rpc error: code = NotFound desc = could not find container \"f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66\": container with ID starting with f7d6c622640966b96f6b0ddce5a0701c710dab032a0bcb5d9ad629d7db507a66 not found: ID does not exist" Nov 02 14:19:49.451921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.451902 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:19:49.466968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.466941 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-credential-socket\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.467144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467011 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-envoy\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.467144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467072 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-socket\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.467144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467123 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-certs\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.467330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467156 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-token\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-token\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.467330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467194 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-certs\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.467330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467233 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b605f12-2143-4e18-8a4d-87945d237404-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b605f12-2143-4e18-8a4d-87945d237404-istio-podinfo\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.467330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467249 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-credential-socket" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.467330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467262 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-podinfo\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.467595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467379 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-podinfo\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.467595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467389 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-socket" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.467595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467410 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-certs" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.467595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467424 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-socket\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.467595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467539 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-istio-token\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-istio-token\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.467595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467551 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-certs" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.467595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467576 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-data\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467607 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-certs\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467651 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-data\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467697 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-socket\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467727 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-kube-api-access-tzs8d\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-kube-api-access-tzs8d\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467766 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-envoy\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467783 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-token\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-token\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467800 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-kube-api-access-2nnsc\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-kube-api-access-2nnsc\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467829 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istiod-ca-cert\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467853 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-kube-api-access-xk24z\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-kube-api-access-xk24z\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467881 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-certs" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467897 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istiod-ca-cert\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467934 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-credential-socket\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467939 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-socket" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.468012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.467969 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b605f12-2143-4e18-8a4d-87945d237404-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b605f12-2143-4e18-8a4d-87945d237404-istiod-ca-cert\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468057 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-credential-socket\") pod \"4b605f12-2143-4e18-8a4d-87945d237404\" (UID: \"4b605f12-2143-4e18-8a4d-87945d237404\") " Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468126 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-envoy\") pod \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\" (UID: \"f3cfb3f5-5903-47dc-9837-27c0f18adabb\") " Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468153 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-data" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468159 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-data" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468179 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-data\") pod \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\" (UID: \"27d9e0b3-69fc-4536-b3e2-be7e8d6714be\") " Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468366 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-credential-socket" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468413 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-credential-socket" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468452 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istiod-ca-cert" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:49.468776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.468609 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b605f12-2143-4e18-8a4d-87945d237404-istiod-ca-cert" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469102 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469126 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469142 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469156 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469170 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469185 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc55x\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-kube-api-access-fc55x\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469200 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469214 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469227 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469240 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b605f12-2143-4e18-8a4d-87945d237404-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469256 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469270 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469285 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469299 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469307 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-socket" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.469346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469314 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.471131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469362 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2465170c-9935-4a55-a00d-7dcaeeb78338-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.471131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469382 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.471131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469397 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.471131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469412 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.471131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469427 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2465170c-9935-4a55-a00d-7dcaeeb78338-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.471131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.469407 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istiod-ca-cert" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:49.471131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.470023 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-data" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.471131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.470971 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b605f12-2143-4e18-8a4d-87945d237404-istio-podinfo" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:19:49.472051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.471967 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-podinfo" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:19:49.472051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.472004 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-envoy" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.472223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.472123 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-kube-api-access-xk24z" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "kube-api-access-xk24z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:49.472223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.472146 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-token" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:49.472223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.472199 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-envoy" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.472325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.472294 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-token" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:49.472458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.472432 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-istio-token" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:49.472793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.472768 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-kube-api-access-tzs8d" pod "4b605f12-2143-4e18-8a4d-87945d237404" (UID: "4b605f12-2143-4e18-8a4d-87945d237404"). InnerVolumeSpecName "kube-api-access-tzs8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:49.472793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.472787 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-envoy" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:19:49.473817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.473794 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-kube-api-access-2nnsc" pod "f3cfb3f5-5903-47dc-9837-27c0f18adabb" (UID: "f3cfb3f5-5903-47dc-9837-27c0f18adabb"). InnerVolumeSpecName "kube-api-access-2nnsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:49.473897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.473841 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-podinfo" pod "27d9e0b3-69fc-4536-b3e2-be7e8d6714be" (UID: "27d9e0b3-69fc-4536-b3e2-be7e8d6714be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:19:49.480641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.480618 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s"] Nov 02 14:19:49.483634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.483613 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-xmd9s"] Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.570905 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.570947 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.570957 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b605f12-2143-4e18-8a4d-87945d237404-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.570967 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.570977 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.570998 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571007 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571017 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzs8d\" (UniqueName: \"kubernetes.io/projected/4b605f12-2143-4e18-8a4d-87945d237404-kube-api-access-tzs8d\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571025 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b605f12-2143-4e18-8a4d-87945d237404-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571033 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571041 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nnsc\" (UniqueName: \"kubernetes.io/projected/f3cfb3f5-5903-47dc-9837-27c0f18adabb-kube-api-access-2nnsc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571049 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk24z\" (UniqueName: \"kubernetes.io/projected/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-kube-api-access-xk24z\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571058 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571067 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cfb3f5-5903-47dc-9837-27c0f18adabb-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:49.571434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:49.571074 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d9e0b3-69fc-4536-b3e2-be7e8d6714be-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:19:50.436841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.436808 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" Nov 02 14:19:50.437349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.436817 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr" event={"ID":"27d9e0b3-69fc-4536-b3e2-be7e8d6714be","Type":"ContainerDied","Data":"bff2e4daf014ce542cf1c08add58d2056ff2340c2a8ea5b2edcf1edcd976ff53"} Nov 02 14:19:50.437349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.436955 5591 scope.go:122] "RemoveContainer" containerID="1d9684187d171283d535e2ab15e9f75ad71f2496534120879eb2b8f296694dc6" Nov 02 14:19:50.439537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.439515 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" Nov 02 14:19:50.439674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.439561 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v" event={"ID":"4b605f12-2143-4e18-8a4d-87945d237404","Type":"ContainerDied","Data":"5368f48358b966dd5d9c30c955b62fa9436accddd7c00106d9ee7a4ffbfda244"} Nov 02 14:19:50.441164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.441137 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" event={"ID":"f3cfb3f5-5903-47dc-9837-27c0f18adabb","Type":"ContainerDied","Data":"5d72e4641b76676fdcfef3bfbf3fa82b28f9394ee5536a1dfe233e4f7e1c16e5"} Nov 02 14:19:50.441302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.441177 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5" Nov 02 14:19:50.450496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.450473 5591 scope.go:122] "RemoveContainer" containerID="2fd38e9a358b501f9b2356e49db93c0601fc7aeba0681bd5beca34e3bb0352af" Nov 02 14:19:50.460142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.460120 5591 scope.go:122] "RemoveContainer" containerID="45192053bf7cb9a2e18f0afb88a9a46df1b9b0266f37afa24614e5ac5e3a7a15" Nov 02 14:19:50.507868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.507840 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr"] Nov 02 14:19:50.520581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.520553 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-sqknr"] Nov 02 14:19:50.547399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.547367 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v"] Nov 02 14:19:50.550971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.550945 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-txm9v"] Nov 02 14:19:50.587468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.587442 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5"] Nov 02 14:19:50.591552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.591529 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-5z8p5"] Nov 02 14:19:50.742122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:50.739382 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:51.454330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.454288 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" path="/var/lib/kubelet/pods/2465170c-9935-4a55-a00d-7dcaeeb78338/volumes" Nov 02 14:19:51.455167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.455135 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" path="/var/lib/kubelet/pods/27d9e0b3-69fc-4536-b3e2-be7e8d6714be/volumes" Nov 02 14:19:51.455897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.455871 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b605f12-2143-4e18-8a4d-87945d237404" path="/var/lib/kubelet/pods/4b605f12-2143-4e18-8a4d-87945d237404/volumes" Nov 02 14:19:51.457033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.457008 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" path="/var/lib/kubelet/pods/f3cfb3f5-5903-47dc-9837-27c0f18adabb/volumes" Nov 02 14:19:51.457475 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.457455 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb"] Nov 02 14:19:51.540290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.540246 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:51.743828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.743749 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7039-61c01b"] Nov 02 14:19:51.744463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744446 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" Nov 02 14:19:51.744524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744465 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" Nov 02 14:19:51.744524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744481 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" Nov 02 14:19:51.744524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744486 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" Nov 02 14:19:51.744524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744498 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" Nov 02 14:19:51.744524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744505 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" Nov 02 14:19:51.744524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744515 5591 cpu_manager.go:414] "containerMap: removing container" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" Nov 02 14:19:51.744524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744520 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" Nov 02 14:19:51.744753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744540 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" Nov 02 14:19:51.744753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744546 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" Nov 02 14:19:51.744753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744643 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2465170c-9935-4a55-a00d-7dcaeeb78338" containerName="istio-proxy" Nov 02 14:19:51.744753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744657 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cfb3f5-5903-47dc-9837-27c0f18adabb" containerName="istio-proxy" Nov 02 14:19:51.744753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744666 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9def5c6-d319-49bd-9a73-3dfdfcd14843" containerName="istio-proxy" Nov 02 14:19:51.744753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744673 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d9e0b3-69fc-4536-b3e2-be7e8d6714be" containerName="istio-proxy" Nov 02 14:19:51.744753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.744682 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b605f12-2143-4e18-8a4d-87945d237404" containerName="istio-proxy" Nov 02 14:19:51.749852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.749832 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-61c01b" Nov 02 14:19:51.765901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.764408 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-61c01b"] Nov 02 14:19:51.766262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.766239 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7jzh4\"" Nov 02 14:19:51.807624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.807587 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5hgx\" (UniqueName: \"kubernetes.io/projected/e661d6ff-af97-4cfb-bee6-0645680d5a43-kube-api-access-p5hgx\") pod \"test-curl-7039-61c01b\" (UID: \"e661d6ff-af97-4cfb-bee6-0645680d5a43\") " pod="opendatahub/test-curl-7039-61c01b" Nov 02 14:19:51.909379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.909345 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hgx\" (UniqueName: \"kubernetes.io/projected/e661d6ff-af97-4cfb-bee6-0645680d5a43-kube-api-access-p5hgx\") pod \"test-curl-7039-61c01b\" (UID: \"e661d6ff-af97-4cfb-bee6-0645680d5a43\") " pod="opendatahub/test-curl-7039-61c01b" Nov 02 14:19:51.933460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:51.933421 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hgx\" (UniqueName: \"kubernetes.io/projected/e661d6ff-af97-4cfb-bee6-0645680d5a43-kube-api-access-p5hgx\") pod \"test-curl-7039-61c01b\" (UID: \"e661d6ff-af97-4cfb-bee6-0645680d5a43\") " pod="opendatahub/test-curl-7039-61c01b" Nov 02 14:19:52.063259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:52.063175 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-61c01b" Nov 02 14:19:52.385856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:52.385822 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-61c01b"] Nov 02 14:19:52.386254 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:19:52.386226 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode661d6ff_af97_4cfb_bee6_0645680d5a43.slice/crio-b54405b953d5d77d420eb9dcd5b0bf271499b4d181845a683b694536bb07c8b5 WatchSource:0}: Error finding container b54405b953d5d77d420eb9dcd5b0bf271499b4d181845a683b694536bb07c8b5: Status 404 returned error can't find the container with id b54405b953d5d77d420eb9dcd5b0bf271499b4d181845a683b694536bb07c8b5 Nov 02 14:19:52.489802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:52.489758 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:52.520776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:52.517949 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-61c01b" event={"ID":"e661d6ff-af97-4cfb-bee6-0645680d5a43","Type":"ContainerStarted","Data":"b54405b953d5d77d420eb9dcd5b0bf271499b4d181845a683b694536bb07c8b5"} Nov 02 14:19:53.527568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:53.527530 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-61c01b" event={"ID":"e661d6ff-af97-4cfb-bee6-0645680d5a43","Type":"ContainerStarted","Data":"3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854"} Nov 02 14:19:53.574618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:53.574557 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7039-61c01b" podStartSLOduration=2.57452994 podStartE2EDuration="2.57452994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:53.571654014 +0000 UTC m=+2142.297275784" watchObservedRunningTime="2026-11-02 14:19:53.57452994 +0000 UTC m=+2142.300151709" Nov 02 14:19:54.207005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:54.204885 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:55.984087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:55.984038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:57.744743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:57.744710 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:19:59.344003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:19:59.342726 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:01.097019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:01.086553 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:01.124384 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:01.124343 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:20:02.656006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:02.653646 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:04.265022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:04.259538 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:05.830275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:05.830237 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:06.035246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.035212 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn"] Nov 02 14:20:06.047363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.047324 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19854b08-m8dvn"] Nov 02 14:20:06.481810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.481743 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" podUID="4abf5d52-3331-44ea-a5b1-71509bb3fc5c" containerName="main" containerID="cri-o://40890d5d9d297a45071b00b7ec98e42dd5c384181effddf9c6e817bee102b321" gracePeriod=45 Nov 02 14:20:06.615213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.615181 5591 generic.go:356] "Generic (PLEG): container finished" podID="4abf5d52-3331-44ea-a5b1-71509bb3fc5c" containerID="40890d5d9d297a45071b00b7ec98e42dd5c384181effddf9c6e817bee102b321" exitCode=0 Nov 02 14:20:06.615386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.615264 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" event={"ID":"4abf5d52-3331-44ea-a5b1-71509bb3fc5c","Type":"ContainerDied","Data":"40890d5d9d297a45071b00b7ec98e42dd5c384181effddf9c6e817bee102b321"} Nov 02 14:20:06.748240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.748181 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:20:06.785802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.785774 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-dshm\") pod \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " Nov 02 14:20:06.785946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.785855 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-model-cache\") pod \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " Nov 02 14:20:06.785946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.785876 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-kube-api-access-g9q4m\" (UniqueName: \"kubernetes.io/projected/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-kube-api-access-g9q4m\") pod \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " Nov 02 14:20:06.786051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.785952 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tmp-dir\") pod \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " Nov 02 14:20:06.786051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.786036 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-home\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-home\") pod \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " Nov 02 14:20:06.786170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.786107 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tls-certs\") pod \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\" (UID: \"4abf5d52-3331-44ea-a5b1-71509bb3fc5c\") " Nov 02 14:20:06.786229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.786158 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-model-cache" pod "4abf5d52-3331-44ea-a5b1-71509bb3fc5c" (UID: "4abf5d52-3331-44ea-a5b1-71509bb3fc5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:06.786309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.786270 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tmp-dir" pod "4abf5d52-3331-44ea-a5b1-71509bb3fc5c" (UID: "4abf5d52-3331-44ea-a5b1-71509bb3fc5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:06.786309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.786292 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-home" pod "4abf5d52-3331-44ea-a5b1-71509bb3fc5c" (UID: "4abf5d52-3331-44ea-a5b1-71509bb3fc5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:06.786703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.786680 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:06.786703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.786704 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:06.786874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.786712 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:06.788262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.788229 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-dshm" pod "4abf5d52-3331-44ea-a5b1-71509bb3fc5c" (UID: "4abf5d52-3331-44ea-a5b1-71509bb3fc5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:06.788450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.788427 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tls-certs" pod "4abf5d52-3331-44ea-a5b1-71509bb3fc5c" (UID: "4abf5d52-3331-44ea-a5b1-71509bb3fc5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:20:06.788918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.788895 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-kube-api-access-g9q4m" pod "4abf5d52-3331-44ea-a5b1-71509bb3fc5c" (UID: "4abf5d52-3331-44ea-a5b1-71509bb3fc5c"). InnerVolumeSpecName "kube-api-access-g9q4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:06.887915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.887879 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:06.887915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.887909 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:06.887915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:06.887919 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9q4m\" (UniqueName: \"kubernetes.io/projected/4abf5d52-3331-44ea-a5b1-71509bb3fc5c-kube-api-access-g9q4m\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:07.255911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.255866 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f"] Nov 02 14:20:07.256576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.256559 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4abf5d52-3331-44ea-a5b1-71509bb3fc5c" containerName="main" Nov 02 14:20:07.256631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.256578 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abf5d52-3331-44ea-a5b1-71509bb3fc5c" containerName="main" Nov 02 14:20:07.256683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.256674 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abf5d52-3331-44ea-a5b1-71509bb3fc5c" containerName="main" Nov 02 14:20:07.260301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.260281 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.299903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.296698 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f"] Nov 02 14:20:07.299903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.296808 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a4809a-serving-cert\"" Nov 02 14:20:07.299903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.297869 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzz5\" (UniqueName: \"kubernetes.io/projected/95c695c6-5acc-481e-a4b0-06c5fe2d933b-kube-api-access-tpzz5\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.299903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.297924 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.299903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.298046 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.299903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.298135 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c695c6-5acc-481e-a4b0-06c5fe2d933b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.399569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.399537 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzz5\" (UniqueName: \"kubernetes.io/projected/95c695c6-5acc-481e-a4b0-06c5fe2d933b-kube-api-access-tpzz5\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.399765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.399578 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.399765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.399608 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.399765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.399654 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c695c6-5acc-481e-a4b0-06c5fe2d933b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.399765 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:07.399739 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-serving-cert: secret "maas-api-e2e-shared-a-w3-a4809a-serving-cert" not found Nov 02 14:20:07.400029 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:07.399810 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls podName:95c695c6-5acc-481e-a4b0-06c5fe2d933b nodeName:}" failed. No retries permitted until 2026-11-02 14:20:07.899792503 +0000 UTC m=+2156.625414272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" (UID: "95c695c6-5acc-481e-a4b0-06c5fe2d933b") : secret "maas-api-e2e-shared-a-w3-a4809a-serving-cert" not found Nov 02 14:20:07.400377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.400357 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c695c6-5acc-481e-a4b0-06c5fe2d933b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.402123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.402101 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.405112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.405088 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:07.425375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.425352 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzz5\" (UniqueName: \"kubernetes.io/projected/95c695c6-5acc-481e-a4b0-06c5fe2d933b-kube-api-access-tpzz5\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.431388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.431361 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0174298a-a6d1-497e-be27-56d96e01e223" path="/var/lib/kubelet/pods/0174298a-a6d1-497e-be27-56d96e01e223/volumes" Nov 02 14:20:07.571088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.570937 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:07.622621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.622586 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" event={"ID":"4abf5d52-3331-44ea-a5b1-71509bb3fc5c","Type":"ContainerDied","Data":"11259a1d605a195ee5cc6d7eb258ce15e4e038fcf1b408310dc642f2b764071b"} Nov 02 14:20:07.622621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.622620 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb" Nov 02 14:20:07.622845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.622643 5591 scope.go:122] "RemoveContainer" containerID="40890d5d9d297a45071b00b7ec98e42dd5c384181effddf9c6e817bee102b321" Nov 02 14:20:07.677931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.677899 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb"] Nov 02 14:20:07.681954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.681926 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/auth-test-model-4c2811-kserve-5ccbcd9f65-mgvrb"] Nov 02 14:20:07.906361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.906321 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:07.908869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:07.908840 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:08.133862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.133823 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk"] Nov 02 14:20:08.134132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.134107 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" containerID="cri-o://a241e61dc0a74015f71286938df5992201485b6a7ceb36318e1428a6c5d8fe02" gracePeriod=30 Nov 02 14:20:08.147437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.147405 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w"] Nov 02 14:20:08.147668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.147644 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" containerID="cri-o://7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8" gracePeriod=30 Nov 02 14:20:08.151477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.151444 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j"] Nov 02 14:20:08.151714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.151684 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" containerID="cri-o://2eb2d6e0c6d3d5e920868f3e5d9b96bcfba26359e03f226548c4792e83b60e67" gracePeriod=30 Nov 02 14:20:08.159960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.159901 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878"] Nov 02 14:20:08.160266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.160240 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" containerID="cri-o://d66717c9e586b1d57dacbe10f166abc08d67420f55dc93c3a6c3cf0a8b4bf1a2" gracePeriod=30 Nov 02 14:20:08.174099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.174066 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:08.580489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.576430 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f"] Nov 02 14:20:08.589016 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:08.587949 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c695c6_5acc_481e_a4b0_06c5fe2d933b.slice/crio-e06ed40d9ea082628c1deda8c97d10f669e41a8137b240c12d05c656d973a8a9 WatchSource:0}: Error finding container e06ed40d9ea082628c1deda8c97d10f669e41a8137b240c12d05c656d973a8a9: Status 404 returned error can't find the container with id e06ed40d9ea082628c1deda8c97d10f669e41a8137b240c12d05c656d973a8a9 Nov 02 14:20:08.590662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.590521 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:20:08.630562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.630532 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" event={"ID":"95c695c6-5acc-481e-a4b0-06c5fe2d933b","Type":"ContainerStarted","Data":"e06ed40d9ea082628c1deda8c97d10f669e41a8137b240c12d05c656d973a8a9"} Nov 02 14:20:08.765320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.765287 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7039-61c01b"] Nov 02 14:20:08.765495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.765461 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7039-61c01b" podUID="e661d6ff-af97-4cfb-bee6-0645680d5a43" containerName="test-curl-7039-61c01b" containerID="cri-o://3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854" gracePeriod=2 Nov 02 14:20:08.783531 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:08.783486 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-61c01b\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="e661d6ff-af97-4cfb-bee6-0645680d5a43" pod="opendatahub/test-curl-7039-61c01b" Nov 02 14:20:08.785449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:08.785423 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7039-61c01b"] Nov 02 14:20:09.444017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:09.443864 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abf5d52-3331-44ea-a5b1-71509bb3fc5c" path="/var/lib/kubelet/pods/4abf5d52-3331-44ea-a5b1-71509bb3fc5c/volumes" Nov 02 14:20:09.472536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:09.472483 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:09.640637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:09.640596 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" event={"ID":"95c695c6-5acc-481e-a4b0-06c5fe2d933b","Type":"ContainerStarted","Data":"e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e"} Nov 02 14:20:09.640792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:09.640667 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:09.692766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:09.692701 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" podStartSLOduration=2.202303577 podStartE2EDuration="2.692682837s" totalImagesPullingTime="490.37926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:09.688723167 +0000 UTC m=+2158.414344938" watchObservedRunningTime="2026-11-02 14:20:09.692682837 +0000 UTC m=+2158.418304607" Nov 02 14:20:10.946942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:10.946914 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7039-825bc1"] Nov 02 14:20:10.947572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:10.947553 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e661d6ff-af97-4cfb-bee6-0645680d5a43" containerName="test-curl-7039-61c01b" Nov 02 14:20:10.947619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:10.947573 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e661d6ff-af97-4cfb-bee6-0645680d5a43" containerName="test-curl-7039-61c01b" Nov 02 14:20:10.947715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:10.947706 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e661d6ff-af97-4cfb-bee6-0645680d5a43" containerName="test-curl-7039-61c01b" Nov 02 14:20:10.951246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:10.951228 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:10.967935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:10.967901 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-825bc1"] Nov 02 14:20:10.982357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:10.982265 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mck\" (UniqueName: \"kubernetes.io/projected/68da37ee-941b-4ee5-b11f-fbdd91603d9e-kube-api-access-d7mck\") pod \"test-curl-7039-825bc1\" (UID: \"68da37ee-941b-4ee5-b11f-fbdd91603d9e\") " pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:10.982527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:10.982391 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-61c01b" Nov 02 14:20:11.084037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.083943 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e661d6ff-af97-4cfb-bee6-0645680d5a43-kube-api-access-p5hgx\" (UniqueName: \"kubernetes.io/projected/e661d6ff-af97-4cfb-bee6-0645680d5a43-kube-api-access-p5hgx\") pod \"e661d6ff-af97-4cfb-bee6-0645680d5a43\" (UID: \"e661d6ff-af97-4cfb-bee6-0645680d5a43\") " Nov 02 14:20:11.084285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.084264 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mck\" (UniqueName: \"kubernetes.io/projected/68da37ee-941b-4ee5-b11f-fbdd91603d9e-kube-api-access-d7mck\") pod \"test-curl-7039-825bc1\" (UID: \"68da37ee-941b-4ee5-b11f-fbdd91603d9e\") " pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:11.086187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.086160 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e661d6ff-af97-4cfb-bee6-0645680d5a43-kube-api-access-p5hgx" pod "e661d6ff-af97-4cfb-bee6-0645680d5a43" (UID: "e661d6ff-af97-4cfb-bee6-0645680d5a43"). InnerVolumeSpecName "kube-api-access-p5hgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:11.109419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.109386 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mck\" (UniqueName: \"kubernetes.io/projected/68da37ee-941b-4ee5-b11f-fbdd91603d9e-kube-api-access-d7mck\") pod \"test-curl-7039-825bc1\" (UID: \"68da37ee-941b-4ee5-b11f-fbdd91603d9e\") " pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:11.185695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.185654 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5hgx\" (UniqueName: \"kubernetes.io/projected/e661d6ff-af97-4cfb-bee6-0645680d5a43-kube-api-access-p5hgx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:11.303407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.293700 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:11.303407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.301960 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:11.471035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.470317 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e661d6ff-af97-4cfb-bee6-0645680d5a43" path="/var/lib/kubelet/pods/e661d6ff-af97-4cfb-bee6-0645680d5a43/volumes" Nov 02 14:20:11.650685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.650649 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-825bc1"] Nov 02 14:20:11.651175 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:11.651146 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68da37ee_941b_4ee5_b11f_fbdd91603d9e.slice/crio-7e97294043d8520d3aa85e46417ab5161cf0763dbc3f80f3f5742bb9f0ea9380 WatchSource:0}: Error finding container 7e97294043d8520d3aa85e46417ab5161cf0763dbc3f80f3f5742bb9f0ea9380: Status 404 returned error can't find the container with id 7e97294043d8520d3aa85e46417ab5161cf0763dbc3f80f3f5742bb9f0ea9380 Nov 02 14:20:11.656059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.656026 5591 generic.go:356] "Generic (PLEG): container finished" podID="e661d6ff-af97-4cfb-bee6-0645680d5a43" containerID="3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854" exitCode=137 Nov 02 14:20:11.656119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.656099 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-61c01b" Nov 02 14:20:11.656158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.656131 5591 scope.go:122] "RemoveContainer" containerID="3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854" Nov 02 14:20:11.667176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.667140 5591 scope.go:122] "RemoveContainer" containerID="3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854" Nov 02 14:20:11.667504 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:11.667481 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854\": container with ID starting with 3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854 not found: ID does not exist" containerID="3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854" Nov 02 14:20:11.667599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:11.667518 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854"} err="failed to get container status \"3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854\": rpc error: code = NotFound desc = could not find container \"3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854\": container with ID starting with 3853ed5ad07e5664101d92ba1563baa6a9b5a6ed7e6795714fbf9acad9b5a854 not found: ID does not exist" Nov 02 14:20:12.580882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:12.580847 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:20:12.666888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:12.666848 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-825bc1" event={"ID":"68da37ee-941b-4ee5-b11f-fbdd91603d9e","Type":"ContainerStarted","Data":"4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45"} Nov 02 14:20:12.667120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:12.666897 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-825bc1" event={"ID":"68da37ee-941b-4ee5-b11f-fbdd91603d9e","Type":"ContainerStarted","Data":"7e97294043d8520d3aa85e46417ab5161cf0763dbc3f80f3f5742bb9f0ea9380"} Nov 02 14:20:12.709766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:12.709713 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7039-825bc1" podStartSLOduration=2.70969793 podStartE2EDuration="2.70969793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:12.706439666 +0000 UTC m=+2161.432061646" watchObservedRunningTime="2026-11-02 14:20:12.70969793 +0000 UTC m=+2161.435319700" Nov 02 14:20:13.004759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:12.999514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:13.672976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.672928 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:20:13.699332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.699277 5591 generic.go:356] "Generic (PLEG): container finished" podID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerID="d66717c9e586b1d57dacbe10f166abc08d67420f55dc93c3a6c3cf0a8b4bf1a2" exitCode=0 Nov 02 14:20:13.699523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.699501 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" event={"ID":"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd","Type":"ContainerDied","Data":"d66717c9e586b1d57dacbe10f166abc08d67420f55dc93c3a6c3cf0a8b4bf1a2"} Nov 02 14:20:13.711011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.702956 5591 generic.go:356] "Generic (PLEG): container finished" podID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerID="2eb2d6e0c6d3d5e920868f3e5d9b96bcfba26359e03f226548c4792e83b60e67" exitCode=0 Nov 02 14:20:13.711011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.703164 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" event={"ID":"df6c7bf0-4b50-459f-9265-3ef9336f2e03","Type":"ContainerDied","Data":"2eb2d6e0c6d3d5e920868f3e5d9b96bcfba26359e03f226548c4792e83b60e67"} Nov 02 14:20:13.711011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.708593 5591 generic.go:356] "Generic (PLEG): container finished" podID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerID="7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8" exitCode=0 Nov 02 14:20:13.711011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.708682 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" event={"ID":"47c22a06-aa23-4cf5-a58f-c75ae82225b5","Type":"ContainerDied","Data":"7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8"} Nov 02 14:20:13.711011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.708729 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" event={"ID":"47c22a06-aa23-4cf5-a58f-c75ae82225b5","Type":"ContainerDied","Data":"1f0cb08a6aa11752de751e21372fcd19d4d9a0a66b3f369a1bd1bfe05c4b396d"} Nov 02 14:20:13.711011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.708761 5591 scope.go:122] "RemoveContainer" containerID="7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8" Nov 02 14:20:13.711011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.708870 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w" Nov 02 14:20:13.719097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.717049 5591 generic.go:356] "Generic (PLEG): container finished" podID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerID="a241e61dc0a74015f71286938df5992201485b6a7ceb36318e1428a6c5d8fe02" exitCode=0 Nov 02 14:20:13.719097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.717697 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" event={"ID":"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3","Type":"ContainerDied","Data":"a241e61dc0a74015f71286938df5992201485b6a7ceb36318e1428a6c5d8fe02"} Nov 02 14:20:13.732482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.731851 5591 scope.go:122] "RemoveContainer" containerID="7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8" Nov 02 14:20:13.732482 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:13.732313 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8\": container with ID starting with 7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8 not found: ID does not exist" containerID="7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8" Nov 02 14:20:13.732482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.732350 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8"} err="failed to get container status \"7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8\": rpc error: code = NotFound desc = could not find container \"7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8\": container with ID starting with 7ec8aadeb5dc2fdcabb79fa4f89e8d083a0d5d41f3fe2c97dd737f3d1f2189b8 not found: ID does not exist" Nov 02 14:20:13.763004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.762880 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-certs\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.763004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.762962 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-data\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.764232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.763255 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-token\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-token\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.764232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.763340 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-socket\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.764232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.763390 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istiod-ca-cert\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.764232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.763445 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-envoy\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.764232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.763506 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-kube-api-access-xv5l7\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-kube-api-access-xv5l7\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.764232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.763535 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-credential-socket\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.764232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.763611 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-podinfo\") pod \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\" (UID: \"47c22a06-aa23-4cf5-a58f-c75ae82225b5\") " Nov 02 14:20:13.777384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.774398 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-socket" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.777384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.775916 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-data" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.777384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.776415 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-certs" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.783259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.783225 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-credential-socket" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.783605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.783553 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istiod-ca-cert" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:13.789163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.789128 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-podinfo" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:20:13.819178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.815077 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:20:13.819178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.816779 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-token" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:13.819178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.818192 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-envoy" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.826688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.826261 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-kube-api-access-xv5l7" pod "47c22a06-aa23-4cf5-a58f-c75ae82225b5" (UID: "47c22a06-aa23-4cf5-a58f-c75ae82225b5"). InnerVolumeSpecName "kube-api-access-xv5l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.869928 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-credential-socket\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.870068 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-kube-api-access-nwc9l\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-kube-api-access-nwc9l\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.870101 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istiod-ca-cert\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.870173 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-podinfo\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.870214 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-envoy\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.870245 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-certs\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.870346 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-socket\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.870473 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-data\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.870506 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-token\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-token\") pod \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\" (UID: \"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3\") " Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871255 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871275 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871288 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871301 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871324 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871337 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871350 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv5l7\" (UniqueName: \"kubernetes.io/projected/47c22a06-aa23-4cf5-a58f-c75ae82225b5-kube-api-access-xv5l7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871364 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c22a06-aa23-4cf5-a58f-c75ae82225b5-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871388 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c22a06-aa23-4cf5-a58f-c75ae82225b5-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.871946 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-credential-socket" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.878688 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-kube-api-access-nwc9l" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "kube-api-access-nwc9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.879102 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istiod-ca-cert" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:13.883065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.881714 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-podinfo" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:20:13.886933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.883949 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-envoy" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.886933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.884196 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-certs" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.886933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.884426 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-socket" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.886933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.884613 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-data" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.890865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.890668 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-token" pod "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" (UID: "ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:13.899393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.899075 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 14:20:13.901255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.901229 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7039-825bc1"] Nov 02 14:20:13.919812 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.919500 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7039-825bc1"] Nov 02 14:20:13.972780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.972744 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-kube-api-access-hm6x7\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-kube-api-access-hm6x7\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.972995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.972813 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-data\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.972995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.972888 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-credential-socket\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.972995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.972917 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-envoy\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.972995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.972953 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-socket\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.973248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.973089 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-token\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-token\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.973248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.973197 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istiod-ca-cert\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.973377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.973311 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-certs\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.973435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.973390 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-podinfo\") pod \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\" (UID: \"df6c7bf0-4b50-459f-9265-3ef9336f2e03\") " Nov 02 14:20:13.974130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974102 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.974130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974131 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwc9l\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-kube-api-access-nwc9l\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.974324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974146 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.974324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974161 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.974324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974173 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.974324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974188 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.974324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974201 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.974324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974213 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.974324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.974227 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:13.976856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.976810 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-token" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:13.977014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.976907 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-credential-socket" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.977115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.976918 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-data" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.977180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.977136 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-socket" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.977247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.977202 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-certs" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.977361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.977331 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-podinfo" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:20:13.977857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.977825 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istiod-ca-cert" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:13.978700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.978624 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-envoy" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:13.979105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:13.979085 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-kube-api-access-hm6x7" pod "df6c7bf0-4b50-459f-9265-3ef9336f2e03" (UID: "df6c7bf0-4b50-459f-9265-3ef9336f2e03"). InnerVolumeSpecName "kube-api-access-hm6x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:14.033048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.033022 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:20:14.066060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.066029 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w"] Nov 02 14:20:14.073192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.073128 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-tm76w"] Nov 02 14:20:14.076415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076390 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.076415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076416 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.076573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076424 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.076573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076434 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.076573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076442 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6x7\" (UniqueName: \"kubernetes.io/projected/df6c7bf0-4b50-459f-9265-3ef9336f2e03-kube-api-access-hm6x7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.076573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076452 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.076573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076461 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.076573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076471 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.076573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.076479 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c7bf0-4b50-459f-9265-3ef9336f2e03-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.080909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.080883 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7039-c47468"] Nov 02 14:20:14.081549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081528 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" Nov 02 14:20:14.081633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081549 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" Nov 02 14:20:14.081633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081577 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" Nov 02 14:20:14.081633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081586 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" Nov 02 14:20:14.081633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081613 5591 cpu_manager.go:414] "containerMap: removing container" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" Nov 02 14:20:14.081633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081620 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" Nov 02 14:20:14.081897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081639 5591 cpu_manager.go:414] "containerMap: removing container" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" Nov 02 14:20:14.081897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081647 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" Nov 02 14:20:14.081897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081706 5591 cpu_manager.go:414] "containerMap: removing container" podUID="68da37ee-941b-4ee5-b11f-fbdd91603d9e" containerName="test-curl-7039-825bc1" Nov 02 14:20:14.081897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081714 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68da37ee-941b-4ee5-b11f-fbdd91603d9e" containerName="test-curl-7039-825bc1" Nov 02 14:20:14.081897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081873 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="68da37ee-941b-4ee5-b11f-fbdd91603d9e" containerName="test-curl-7039-825bc1" Nov 02 14:20:14.081897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081888 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" containerName="istio-proxy" Nov 02 14:20:14.081897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081898 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" containerName="istio-proxy" Nov 02 14:20:14.082272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081910 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" containerName="istio-proxy" Nov 02 14:20:14.082272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.081922 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" containerName="istio-proxy" Nov 02 14:20:14.085612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.085593 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:14.106544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.106514 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-c47468"] Nov 02 14:20:14.177974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.177935 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-data\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178014 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-envoy\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178112 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istiod-ca-cert\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178139 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-podinfo\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178183 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-socket\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178239 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-certs\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178269 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-kube-api-access-f27h4\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-kube-api-access-f27h4\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178312 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-token\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-token\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178234 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-data" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:14.178537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178460 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-credential-socket\") pod \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\" (UID: \"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd\") " Nov 02 14:20:14.178537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178522 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-certs" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:14.178707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178548 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istiod-ca-cert" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:14.178790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.178754 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-credential-socket" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:14.179070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.179004 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-socket" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:14.179271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.179113 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdg25\" (UniqueName: \"kubernetes.io/projected/ebbfa802-5aca-4361-9046-b7b4809f0ab2-kube-api-access-mdg25\") pod \"test-curl-7039-c47468\" (UID: \"ebbfa802-5aca-4361-9046-b7b4809f0ab2\") " pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:14.179405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.179386 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.179476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.179409 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.179476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.179423 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.179476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.179438 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.179476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.179457 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.180643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.180624 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-token" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:14.180798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.180777 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-podinfo" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:20:14.180798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.180784 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-kube-api-access-f27h4" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "kube-api-access-f27h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:14.181018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.180941 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-envoy" pod "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" (UID: "3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:14.281307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.281269 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdg25\" (UniqueName: \"kubernetes.io/projected/ebbfa802-5aca-4361-9046-b7b4809f0ab2-kube-api-access-mdg25\") pod \"test-curl-7039-c47468\" (UID: \"ebbfa802-5aca-4361-9046-b7b4809f0ab2\") " pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:14.281508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.281496 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.281563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.281510 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.281563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.281520 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f27h4\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-kube-api-access-f27h4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.281563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.281529 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:14.305287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.305258 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdg25\" (UniqueName: \"kubernetes.io/projected/ebbfa802-5aca-4361-9046-b7b4809f0ab2-kube-api-access-mdg25\") pod \"test-curl-7039-c47468\" (UID: \"ebbfa802-5aca-4361-9046-b7b4809f0ab2\") " pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:14.398957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.398919 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:14.745070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.735592 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-c47468"] Nov 02 14:20:14.745070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.740512 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:14.765955 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:14.765215 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebbfa802_5aca_4361_9046_b7b4809f0ab2.slice/crio-53ebe11b5af97b02faa511d8b7ce6b4bad0a377e71e5e2480dd8d1834bb8c8de WatchSource:0}: Error finding container 53ebe11b5af97b02faa511d8b7ce6b4bad0a377e71e5e2480dd8d1834bb8c8de: Status 404 returned error can't find the container with id 53ebe11b5af97b02faa511d8b7ce6b4bad0a377e71e5e2480dd8d1834bb8c8de Nov 02 14:20:14.767138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.767100 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" event={"ID":"ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3","Type":"ContainerDied","Data":"f1d91e8d7a468692305bea9a111edeb9d0802da8cd81d61b92c3d37ea71aa8dc"} Nov 02 14:20:14.767400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.767381 5591 scope.go:122] "RemoveContainer" containerID="a241e61dc0a74015f71286938df5992201485b6a7ceb36318e1428a6c5d8fe02" Nov 02 14:20:14.767694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.767679 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk" Nov 02 14:20:14.777958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.777410 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" Nov 02 14:20:14.777958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.777421 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878" event={"ID":"3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd","Type":"ContainerDied","Data":"041f02e31d4ae07ed580fad58be1dd4920cb2bb1c3fbb915e0533a1c26dd6743"} Nov 02 14:20:14.780436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.780416 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" Nov 02 14:20:14.780531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.780431 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j" event={"ID":"df6c7bf0-4b50-459f-9265-3ef9336f2e03","Type":"ContainerDied","Data":"ce02aaf13c296d12d70c32305d06e74df7131743f2f76db4c275c144dafce534"} Nov 02 14:20:14.780647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.780622 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7039-825bc1" podUID="68da37ee-941b-4ee5-b11f-fbdd91603d9e" containerName="test-curl-7039-825bc1" containerID="cri-o://4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45" gracePeriod=2 Nov 02 14:20:14.794500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.794481 5591 scope.go:122] "RemoveContainer" containerID="d66717c9e586b1d57dacbe10f166abc08d67420f55dc93c3a6c3cf0a8b4bf1a2" Nov 02 14:20:14.808769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.808749 5591 scope.go:122] "RemoveContainer" containerID="2eb2d6e0c6d3d5e920868f3e5d9b96bcfba26359e03f226548c4792e83b60e67" Nov 02 14:20:14.809871 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:14.809844 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-825bc1\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="68da37ee-941b-4ee5-b11f-fbdd91603d9e" pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:14.882260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.882228 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk"] Nov 02 14:20:14.884955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.884929 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-fq9dk"] Nov 02 14:20:14.895973 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:14.895944 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-825bc1\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="68da37ee-941b-4ee5-b11f-fbdd91603d9e" pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:14.935618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.935581 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878"] Nov 02 14:20:14.939156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.939126 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-6p878"] Nov 02 14:20:14.975277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.975240 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j"] Nov 02 14:20:14.985596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:14.985569 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rhg5j"] Nov 02 14:20:15.431884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:15.431846 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd" path="/var/lib/kubelet/pods/3ac04a52-c885-4b1f-b8c7-c21fe8e3d4bd/volumes" Nov 02 14:20:15.432558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:15.432540 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c22a06-aa23-4cf5-a58f-c75ae82225b5" path="/var/lib/kubelet/pods/47c22a06-aa23-4cf5-a58f-c75ae82225b5/volumes" Nov 02 14:20:15.433155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:15.433139 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3" path="/var/lib/kubelet/pods/ba0349d3-72dc-4a9e-9fb6-30a622dbb3c3/volumes" Nov 02 14:20:15.434223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:15.434205 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df6c7bf0-4b50-459f-9265-3ef9336f2e03" path="/var/lib/kubelet/pods/df6c7bf0-4b50-459f-9265-3ef9336f2e03/volumes" Nov 02 14:20:15.653047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:15.653017 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:20:15.791468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:15.791366 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-c47468" event={"ID":"ebbfa802-5aca-4361-9046-b7b4809f0ab2","Type":"ContainerStarted","Data":"14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061"} Nov 02 14:20:15.791468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:15.791417 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-c47468" event={"ID":"ebbfa802-5aca-4361-9046-b7b4809f0ab2","Type":"ContainerStarted","Data":"53ebe11b5af97b02faa511d8b7ce6b4bad0a377e71e5e2480dd8d1834bb8c8de"} Nov 02 14:20:15.833209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:15.833159 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7039-c47468" podStartSLOduration=1.833144364 podStartE2EDuration="1.833144364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:15.830431634 +0000 UTC m=+2164.556053404" watchObservedRunningTime="2026-11-02 14:20:15.833144364 +0000 UTC m=+2164.558766133" Nov 02 14:20:16.676009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.671113 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:16.802715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.802690 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7039-c47468"] Nov 02 14:20:16.806212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.806187 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7039-c47468"] Nov 02 14:20:16.973486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.973403 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7039-bb26ba"] Nov 02 14:20:16.974134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.974116 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" containerName="test-curl-7039-c47468" Nov 02 14:20:16.974134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.974134 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" containerName="test-curl-7039-c47468" Nov 02 14:20:16.974318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.974298 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" containerName="test-curl-7039-c47468" Nov 02 14:20:16.978126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.978102 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:16.986489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:16.986465 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-bb26ba"] Nov 02 14:20:17.004410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.004378 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:17.121155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.121117 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68da37ee-941b-4ee5-b11f-fbdd91603d9e-kube-api-access-d7mck\" (UniqueName: \"kubernetes.io/projected/68da37ee-941b-4ee5-b11f-fbdd91603d9e-kube-api-access-d7mck\") pod \"68da37ee-941b-4ee5-b11f-fbdd91603d9e\" (UID: \"68da37ee-941b-4ee5-b11f-fbdd91603d9e\") " Nov 02 14:20:17.121384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.121368 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvr6h\" (UniqueName: \"kubernetes.io/projected/a472625b-3fa1-4de1-aa8d-845a7c5b8af0-kube-api-access-pvr6h\") pod \"test-curl-7039-bb26ba\" (UID: \"a472625b-3fa1-4de1-aa8d-845a7c5b8af0\") " pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:17.123257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.123236 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68da37ee-941b-4ee5-b11f-fbdd91603d9e-kube-api-access-d7mck" pod "68da37ee-941b-4ee5-b11f-fbdd91603d9e" (UID: "68da37ee-941b-4ee5-b11f-fbdd91603d9e"). InnerVolumeSpecName "kube-api-access-d7mck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:17.222630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.222585 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvr6h\" (UniqueName: \"kubernetes.io/projected/a472625b-3fa1-4de1-aa8d-845a7c5b8af0-kube-api-access-pvr6h\") pod \"test-curl-7039-bb26ba\" (UID: \"a472625b-3fa1-4de1-aa8d-845a7c5b8af0\") " pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:17.222888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.222813 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mck\" (UniqueName: \"kubernetes.io/projected/68da37ee-941b-4ee5-b11f-fbdd91603d9e-kube-api-access-d7mck\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:17.247021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.246943 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvr6h\" (UniqueName: \"kubernetes.io/projected/a472625b-3fa1-4de1-aa8d-845a7c5b8af0-kube-api-access-pvr6h\") pod \"test-curl-7039-bb26ba\" (UID: \"a472625b-3fa1-4de1-aa8d-845a7c5b8af0\") " pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:17.313877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.313841 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:17.436038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.435908 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68da37ee-941b-4ee5-b11f-fbdd91603d9e" path="/var/lib/kubelet/pods/68da37ee-941b-4ee5-b11f-fbdd91603d9e/volumes" Nov 02 14:20:17.618451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.618420 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-bb26ba"] Nov 02 14:20:17.620119 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:17.620094 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda472625b_3fa1_4de1_aa8d_845a7c5b8af0.slice/crio-0b273f74018eccfb22d4125bf2dc6720d54ed8d190dc076e71f8431482c8fa98 WatchSource:0}: Error finding container 0b273f74018eccfb22d4125bf2dc6720d54ed8d190dc076e71f8431482c8fa98: Status 404 returned error can't find the container with id 0b273f74018eccfb22d4125bf2dc6720d54ed8d190dc076e71f8431482c8fa98 Nov 02 14:20:17.807608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.807568 5591 generic.go:356] "Generic (PLEG): container finished" podID="68da37ee-941b-4ee5-b11f-fbdd91603d9e" containerID="4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45" exitCode=137 Nov 02 14:20:17.808007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.807624 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-825bc1" Nov 02 14:20:17.808007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.807665 5591 scope.go:122] "RemoveContainer" containerID="4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45" Nov 02 14:20:17.809449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.809402 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-bb26ba" event={"ID":"a472625b-3fa1-4de1-aa8d-845a7c5b8af0","Type":"ContainerStarted","Data":"b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb"} Nov 02 14:20:17.809564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.809448 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7039-c47468" podUID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" containerName="test-curl-7039-c47468" containerID="cri-o://14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061" gracePeriod=2 Nov 02 14:20:17.809824 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.809799 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-bb26ba" event={"ID":"a472625b-3fa1-4de1-aa8d-845a7c5b8af0","Type":"ContainerStarted","Data":"0b273f74018eccfb22d4125bf2dc6720d54ed8d190dc076e71f8431482c8fa98"} Nov 02 14:20:17.817344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.817320 5591 scope.go:122] "RemoveContainer" containerID="4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45" Nov 02 14:20:17.817637 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:17.817618 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45\": container with ID starting with 4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45 not found: ID does not exist" containerID="4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45" Nov 02 14:20:17.817689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.817644 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45"} err="failed to get container status \"4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45\": rpc error: code = NotFound desc = could not find container \"4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45\": container with ID starting with 4f326d556405bcced80866b3f13651cab0e61be38f1a9a960f3e79dc19885d45 not found: ID does not exist" Nov 02 14:20:17.840837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.840810 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh"] Nov 02 14:20:17.841057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.841024 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" containerID="cri-o://f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971" gracePeriod=30 Nov 02 14:20:17.849783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:17.849726 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7039-bb26ba" podStartSLOduration=1.849709121 podStartE2EDuration="1.849709121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:17.848582533 +0000 UTC m=+2166.574204301" watchObservedRunningTime="2026-11-02 14:20:17.849709121 +0000 UTC m=+2166.575330893" Nov 02 14:20:18.345943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.345897 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:18.747385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.747346 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7039-bb26ba"] Nov 02 14:20:18.750859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.750831 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7039-bb26ba"] Nov 02 14:20:18.928176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.928146 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7039-a2b375"] Nov 02 14:20:18.928754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.928737 5591 cpu_manager.go:414] "containerMap: removing container" podUID="a472625b-3fa1-4de1-aa8d-845a7c5b8af0" containerName="test-curl-7039-bb26ba" Nov 02 14:20:18.928795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.928755 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a472625b-3fa1-4de1-aa8d-845a7c5b8af0" containerName="test-curl-7039-bb26ba" Nov 02 14:20:18.928880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.928870 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a472625b-3fa1-4de1-aa8d-845a7c5b8af0" containerName="test-curl-7039-bb26ba" Nov 02 14:20:18.933682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.933663 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:18.948430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:18.948404 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-a2b375"] Nov 02 14:20:19.053436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.053349 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4bv\" (UniqueName: \"kubernetes.io/projected/c361df19-33ab-4887-a36b-fd976fe08088-kube-api-access-6r4bv\") pod \"test-curl-7039-a2b375\" (UID: \"c361df19-33ab-4887-a36b-fd976fe08088\") " pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:19.154789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.154751 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4bv\" (UniqueName: \"kubernetes.io/projected/c361df19-33ab-4887-a36b-fd976fe08088-kube-api-access-6r4bv\") pod \"test-curl-7039-a2b375\" (UID: \"c361df19-33ab-4887-a36b-fd976fe08088\") " pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:19.178716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.178684 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4bv\" (UniqueName: \"kubernetes.io/projected/c361df19-33ab-4887-a36b-fd976fe08088-kube-api-access-6r4bv\") pod \"test-curl-7039-a2b375\" (UID: \"c361df19-33ab-4887-a36b-fd976fe08088\") " pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:19.246339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.246308 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:19.548370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.548343 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-a2b375"] Nov 02 14:20:19.549959 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:19.549928 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc361df19_33ab_4887_a36b_fd976fe08088.slice/crio-814706bce10f8369f2bb5ac925f33a5176578e3724d62d001695c52f50cb4b8a WatchSource:0}: Error finding container 814706bce10f8369f2bb5ac925f33a5176578e3724d62d001695c52f50cb4b8a: Status 404 returned error can't find the container with id 814706bce10f8369f2bb5ac925f33a5176578e3724d62d001695c52f50cb4b8a Nov 02 14:20:19.824883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.824854 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-a2b375" event={"ID":"c361df19-33ab-4887-a36b-fd976fe08088","Type":"ContainerStarted","Data":"08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b"} Nov 02 14:20:19.825489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.824895 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-a2b375" event={"ID":"c361df19-33ab-4887-a36b-fd976fe08088","Type":"ContainerStarted","Data":"814706bce10f8369f2bb5ac925f33a5176578e3724d62d001695c52f50cb4b8a"} Nov 02 14:20:19.825489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.825102 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7039-bb26ba" podUID="a472625b-3fa1-4de1-aa8d-845a7c5b8af0" containerName="test-curl-7039-bb26ba" containerID="cri-o://b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb" gracePeriod=2 Nov 02 14:20:19.863710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.863667 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7039-a2b375" podStartSLOduration=1.8636540419999998 podStartE2EDuration="1.863654042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:19.863464418 +0000 UTC m=+2168.589086188" watchObservedRunningTime="2026-11-02 14:20:19.863654042 +0000 UTC m=+2168.589275811" Nov 02 14:20:19.997089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:19.997063 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:20.092955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.092800 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:20.173424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.173391 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbfa802-5aca-4361-9046-b7b4809f0ab2-kube-api-access-mdg25\" (UniqueName: \"kubernetes.io/projected/ebbfa802-5aca-4361-9046-b7b4809f0ab2-kube-api-access-mdg25\") pod \"ebbfa802-5aca-4361-9046-b7b4809f0ab2\" (UID: \"ebbfa802-5aca-4361-9046-b7b4809f0ab2\") " Nov 02 14:20:20.175680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.175653 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbfa802-5aca-4361-9046-b7b4809f0ab2-kube-api-access-mdg25" pod "ebbfa802-5aca-4361-9046-b7b4809f0ab2" (UID: "ebbfa802-5aca-4361-9046-b7b4809f0ab2"). InnerVolumeSpecName "kube-api-access-mdg25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:20.275440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.275396 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdg25\" (UniqueName: \"kubernetes.io/projected/ebbfa802-5aca-4361-9046-b7b4809f0ab2-kube-api-access-mdg25\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:20.323028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.322342 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:20.820338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.820294 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7039-a2b375"] Nov 02 14:20:20.827464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.827433 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7039-a2b375"] Nov 02 14:20:20.834847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.834814 5591 generic.go:356] "Generic (PLEG): container finished" podID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" containerID="14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061" exitCode=137 Nov 02 14:20:20.835034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.834867 5591 scope.go:122] "RemoveContainer" containerID="14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061" Nov 02 14:20:20.835034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.834882 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:20.847239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.847219 5591 scope.go:122] "RemoveContainer" containerID="14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061" Nov 02 14:20:20.847565 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:20.847544 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061\": container with ID starting with 14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061 not found: ID does not exist" containerID="14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061" Nov 02 14:20:20.847614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.847574 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061"} err="failed to get container status \"14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061\": rpc error: code = NotFound desc = could not find container \"14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061\": container with ID starting with 14e0ab7fa858a4da8fdbe8421d3fc098a7db2dc86e81a05aa806c96191ce0061 not found: ID does not exist" Nov 02 14:20:20.862269 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:20.858835 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-c47468\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:20.983446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.983413 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7039-d9c3cc"] Nov 02 14:20:20.984152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.984129 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c361df19-33ab-4887-a36b-fd976fe08088" containerName="test-curl-7039-a2b375" Nov 02 14:20:20.984234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.984155 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c361df19-33ab-4887-a36b-fd976fe08088" containerName="test-curl-7039-a2b375" Nov 02 14:20:20.984340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.984324 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c361df19-33ab-4887-a36b-fd976fe08088" containerName="test-curl-7039-a2b375" Nov 02 14:20:20.989236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:20.989217 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-d9c3cc" Nov 02 14:20:21.003838 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:21.003798 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-c47468\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:21.004691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.004666 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-d9c3cc"] Nov 02 14:20:21.097101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.096964 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wspsf\" (UniqueName: \"kubernetes.io/projected/f590a48c-0e64-4bca-8401-fcacb8386fce-kube-api-access-wspsf\") pod \"test-curl-7039-d9c3cc\" (UID: \"f590a48c-0e64-4bca-8401-fcacb8386fce\") " pod="opendatahub/test-curl-7039-d9c3cc" Nov 02 14:20:21.199113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.199072 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspsf\" (UniqueName: \"kubernetes.io/projected/f590a48c-0e64-4bca-8401-fcacb8386fce-kube-api-access-wspsf\") pod \"test-curl-7039-d9c3cc\" (UID: \"f590a48c-0e64-4bca-8401-fcacb8386fce\") " pod="opendatahub/test-curl-7039-d9c3cc" Nov 02 14:20:21.223431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.223395 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspsf\" (UniqueName: \"kubernetes.io/projected/f590a48c-0e64-4bca-8401-fcacb8386fce-kube-api-access-wspsf\") pod \"test-curl-7039-d9c3cc\" (UID: \"f590a48c-0e64-4bca-8401-fcacb8386fce\") " pod="opendatahub/test-curl-7039-d9c3cc" Nov 02 14:20:21.302484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.302446 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-d9c3cc" Nov 02 14:20:21.433685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.433643 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" path="/var/lib/kubelet/pods/ebbfa802-5aca-4361-9046-b7b4809f0ab2/volumes" Nov 02 14:20:21.442806 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:21.442773 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-c47468\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="ebbfa802-5aca-4361-9046-b7b4809f0ab2" pod="opendatahub/test-curl-7039-c47468" Nov 02 14:20:21.605547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.605513 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7039-d9c3cc"] Nov 02 14:20:21.606781 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:21.606726 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf590a48c_0e64_4bca_8401_fcacb8386fce.slice/crio-506e1434c96da9f47fb501f536d3434443d5bc8c6b5a2475f9e15db5e4ad957c WatchSource:0}: Error finding container 506e1434c96da9f47fb501f536d3434443d5bc8c6b5a2475f9e15db5e4ad957c: Status 404 returned error can't find the container with id 506e1434c96da9f47fb501f536d3434443d5bc8c6b5a2475f9e15db5e4ad957c Nov 02 14:20:21.758092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.757930 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:21.843293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.843261 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-d9c3cc" event={"ID":"f590a48c-0e64-4bca-8401-fcacb8386fce","Type":"ContainerStarted","Data":"92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd"} Nov 02 14:20:21.843429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.843310 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7039-d9c3cc" event={"ID":"f590a48c-0e64-4bca-8401-fcacb8386fce","Type":"ContainerStarted","Data":"506e1434c96da9f47fb501f536d3434443d5bc8c6b5a2475f9e15db5e4ad957c"} Nov 02 14:20:21.844538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.844511 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7039-a2b375" podUID="c361df19-33ab-4887-a36b-fd976fe08088" containerName="test-curl-7039-a2b375" containerID="cri-o://08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b" gracePeriod=2 Nov 02 14:20:21.885692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:21.885645 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7039-d9c3cc" podStartSLOduration=1.885630106 podStartE2EDuration="1.885630106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:21.883792925 +0000 UTC m=+2170.609414694" watchObservedRunningTime="2026-11-02 14:20:21.885630106 +0000 UTC m=+2170.611251876" Nov 02 14:20:22.009796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.009767 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:22.111129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.111043 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a472625b-3fa1-4de1-aa8d-845a7c5b8af0-kube-api-access-pvr6h\" (UniqueName: \"kubernetes.io/projected/a472625b-3fa1-4de1-aa8d-845a7c5b8af0-kube-api-access-pvr6h\") pod \"a472625b-3fa1-4de1-aa8d-845a7c5b8af0\" (UID: \"a472625b-3fa1-4de1-aa8d-845a7c5b8af0\") " Nov 02 14:20:22.113181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.113155 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a472625b-3fa1-4de1-aa8d-845a7c5b8af0-kube-api-access-pvr6h" pod "a472625b-3fa1-4de1-aa8d-845a7c5b8af0" (UID: "a472625b-3fa1-4de1-aa8d-845a7c5b8af0"). InnerVolumeSpecName "kube-api-access-pvr6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:22.212643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.212598 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvr6h\" (UniqueName: \"kubernetes.io/projected/a472625b-3fa1-4de1-aa8d-845a7c5b8af0-kube-api-access-pvr6h\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:22.778891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.778799 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7039-d9c3cc"] Nov 02 14:20:22.783656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.783623 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7039-d9c3cc"] Nov 02 14:20:22.853084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.853036 5591 generic.go:356] "Generic (PLEG): container finished" podID="a472625b-3fa1-4de1-aa8d-845a7c5b8af0" containerID="b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb" exitCode=137 Nov 02 14:20:22.853215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.853085 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:22.853215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.853121 5591 scope.go:122] "RemoveContainer" containerID="b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb" Nov 02 14:20:22.918996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.918963 5591 scope.go:122] "RemoveContainer" containerID="b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb" Nov 02 14:20:22.919325 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:22.919295 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb\": container with ID starting with b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb not found: ID does not exist" containerID="b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb" Nov 02 14:20:22.919390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:22.919334 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb"} err="failed to get container status \"b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb\": rpc error: code = NotFound desc = could not find container \"b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb\": container with ID starting with b7c104b05bc11ed829b906d534d28eb16a0d33e63ce0d388c07aecc81a1b94fb not found: ID does not exist" Nov 02 14:20:22.930022 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:22.929964 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-bb26ba\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="a472625b-3fa1-4de1-aa8d-845a7c5b8af0" pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:23.125422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.125379 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:20:23.139825 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:23.139778 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-bb26ba\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="a472625b-3fa1-4de1-aa8d-845a7c5b8af0" pod="opendatahub/test-curl-7039-bb26ba" Nov 02 14:20:23.232332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232295 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-data\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.232535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232424 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-token\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-token\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.232535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232522 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-socket\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.232656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232564 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-data" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:23.232656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232585 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-certs\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.232656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232621 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-credential-socket\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.232813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232713 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-kube-api-access-4jw47\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-kube-api-access-4jw47\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.232813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232755 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istiod-ca-cert\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.232813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232792 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-podinfo\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.232965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232837 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-socket" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:23.232965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232855 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-certs" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:23.232965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232872 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-envoy\") pod \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\" (UID: \"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05\") " Nov 02 14:20:23.233136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.232959 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-credential-socket" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:23.233302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.233279 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istiod-ca-cert" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:23.233522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.233501 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.233627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.233528 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.233627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.233545 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.233627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.233560 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.233627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.233574 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.234837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.234799 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-token" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:23.235116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.235088 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-kube-api-access-4jw47" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "kube-api-access-4jw47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:23.235225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.235147 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-envoy" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:20:23.235269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.235244 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-podinfo" pod "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" (UID: "5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:20:23.335917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.334871 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.335917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.334909 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.335917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.334924 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.335917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.334938 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jw47\" (UniqueName: \"kubernetes.io/projected/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05-kube-api-access-4jw47\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:23.358060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.356777 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:23.431156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.431116 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a472625b-3fa1-4de1-aa8d-845a7c5b8af0" path="/var/lib/kubelet/pods/a472625b-3fa1-4de1-aa8d-845a7c5b8af0/volumes" Nov 02 14:20:23.862261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.862237 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" Nov 02 14:20:23.862261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.862173 5591 generic.go:356] "Generic (PLEG): container finished" podID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerID="f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971" exitCode=0 Nov 02 14:20:23.862444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.862245 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" event={"ID":"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05","Type":"ContainerDied","Data":"f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971"} Nov 02 14:20:23.862444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.862353 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh" event={"ID":"5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05","Type":"ContainerDied","Data":"e08b8a3d90645693248ec7ecfec2a2b2e5c2f3e1affbd576fbcae2242b8b134e"} Nov 02 14:20:23.862444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.862384 5591 scope.go:122] "RemoveContainer" containerID="f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971" Nov 02 14:20:23.862614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.862577 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7039-d9c3cc" podUID="f590a48c-0e64-4bca-8401-fcacb8386fce" containerName="test-curl-7039-d9c3cc" containerID="cri-o://92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd" gracePeriod=2 Nov 02 14:20:23.927077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.927046 5591 scope.go:122] "RemoveContainer" containerID="f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971" Nov 02 14:20:23.927436 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:23.927409 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971\": container with ID starting with f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971 not found: ID does not exist" containerID="f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971" Nov 02 14:20:23.927539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.927444 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971"} err="failed to get container status \"f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971\": rpc error: code = NotFound desc = could not find container \"f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971\": container with ID starting with f6bc3ebcbb831757bbafdedb75ddd7d1500be63417bbfadffd25b2d01a816971 not found: ID does not exist" Nov 02 14:20:23.973915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.973883 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh"] Nov 02 14:20:23.986352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:23.986325 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-wb5bh"] Nov 02 14:20:24.037584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.037556 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:24.043576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.043548 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c361df19-33ab-4887-a36b-fd976fe08088-kube-api-access-6r4bv\" (UniqueName: \"kubernetes.io/projected/c361df19-33ab-4887-a36b-fd976fe08088-kube-api-access-6r4bv\") pod \"c361df19-33ab-4887-a36b-fd976fe08088\" (UID: \"c361df19-33ab-4887-a36b-fd976fe08088\") " Nov 02 14:20:24.045876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.045852 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c361df19-33ab-4887-a36b-fd976fe08088-kube-api-access-6r4bv" pod "c361df19-33ab-4887-a36b-fd976fe08088" (UID: "c361df19-33ab-4887-a36b-fd976fe08088"). InnerVolumeSpecName "kube-api-access-6r4bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:24.144708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.144663 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:24.145115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.144735 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r4bv\" (UniqueName: \"kubernetes.io/projected/c361df19-33ab-4887-a36b-fd976fe08088-kube-api-access-6r4bv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:24.680182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.680151 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:24.872227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.872188 5591 generic.go:356] "Generic (PLEG): container finished" podID="c361df19-33ab-4887-a36b-fd976fe08088" containerID="08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b" exitCode=137 Nov 02 14:20:24.872227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.872225 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:24.872449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.872282 5591 scope.go:122] "RemoveContainer" containerID="08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b" Nov 02 14:20:24.881944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.881926 5591 scope.go:122] "RemoveContainer" containerID="08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b" Nov 02 14:20:24.882270 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:24.882249 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b\": container with ID starting with 08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b not found: ID does not exist" containerID="08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b" Nov 02 14:20:24.882349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:24.882278 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b"} err="failed to get container status \"08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b\": rpc error: code = NotFound desc = could not find container \"08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b\": container with ID starting with 08a6106a9814862ecbcf4a7c65e30eb5e7de319383edb2bd70db794947ccbd1b not found: ID does not exist" Nov 02 14:20:24.895196 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:24.895166 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-a2b375\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="c361df19-33ab-4887-a36b-fd976fe08088" pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:25.013729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.013645 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c"] Nov 02 14:20:25.014375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.014352 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" Nov 02 14:20:25.014501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.014375 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" Nov 02 14:20:25.014501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.014412 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f590a48c-0e64-4bca-8401-fcacb8386fce" containerName="test-curl-7039-d9c3cc" Nov 02 14:20:25.014501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.014421 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f590a48c-0e64-4bca-8401-fcacb8386fce" containerName="test-curl-7039-d9c3cc" Nov 02 14:20:25.014669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.014562 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" containerName="istio-proxy" Nov 02 14:20:25.014669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.014581 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f590a48c-0e64-4bca-8401-fcacb8386fce" containerName="test-curl-7039-d9c3cc" Nov 02 14:20:25.019479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.019455 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.033972 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:25.033941 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-a2b375\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="c361df19-33ab-4887-a36b-fd976fe08088" pod="opendatahub/test-curl-7039-a2b375" Nov 02 14:20:25.039488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.039458 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-26b508-openshift-default-dockercfg-s78c5\"" Nov 02 14:20:25.054889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.054712 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-data\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.055056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.055021 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-podinfo\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.055144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.055058 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istiod-ca-cert\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.055144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.055090 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-envoy\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.055144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.055133 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-socket\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.055296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.055243 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-certs\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.055370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.055300 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ztm\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-kube-api-access-v8ztm\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.055432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.055374 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-credential-socket\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.055432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.055413 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-token\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.071939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.071909 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c"] Nov 02 14:20:25.140924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.139420 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157373 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-certs\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157449 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ztm\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-kube-api-access-v8ztm\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157487 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-credential-socket\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157522 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-token\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157588 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-data\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157653 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-podinfo\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157680 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istiod-ca-cert\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157715 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-envoy\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.157771 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-socket\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.158258 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-socket\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.158517 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-certs\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.159146 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-credential-socket\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.164004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.159646 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-data\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.177044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.169834 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istiod-ca-cert\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.212009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.211583 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ztm\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-kube-api-access-v8ztm\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.216001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.214555 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-podinfo\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.216001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.214868 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-envoy\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.219688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.217204 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-token\") pod \"e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.339294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.339196 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:25.339294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.339199 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:25.434830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.434797 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05" path="/var/lib/kubelet/pods/5cfd2773-c8bc-4ec8-8a02-210b4ff2bb05/volumes" Nov 02 14:20:25.435248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.435232 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c361df19-33ab-4887-a36b-fd976fe08088" path="/var/lib/kubelet/pods/c361df19-33ab-4887-a36b-fd976fe08088/volumes" Nov 02 14:20:25.669194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.669164 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c"] Nov 02 14:20:25.670461 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:25.670436 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39babc9_1aa2_4878_8b21_b98ddd3a7f34.slice/crio-dadc441472f5a87639ac04c1accfa9bfc93803fb26ca5f4e123e8caee4d92103 WatchSource:0}: Error finding container dadc441472f5a87639ac04c1accfa9bfc93803fb26ca5f4e123e8caee4d92103: Status 404 returned error can't find the container with id dadc441472f5a87639ac04c1accfa9bfc93803fb26ca5f4e123e8caee4d92103 Nov 02 14:20:25.673666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.673632 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:20:25.673760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.673717 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:20:25.673821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.673758 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:20:25.880919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.880730 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" event={"ID":"c39babc9-1aa2-4878-8b21-b98ddd3a7f34","Type":"ContainerStarted","Data":"6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407"} Nov 02 14:20:25.880919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.880804 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" event={"ID":"c39babc9-1aa2-4878-8b21-b98ddd3a7f34","Type":"ContainerStarted","Data":"dadc441472f5a87639ac04c1accfa9bfc93803fb26ca5f4e123e8caee4d92103"} Nov 02 14:20:25.931596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:25.931495 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" podStartSLOduration=1.931480407 podStartE2EDuration="1.931480407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:25.929353628 +0000 UTC m=+2174.654975398" watchObservedRunningTime="2026-11-02 14:20:25.931480407 +0000 UTC m=+2174.657102178" Nov 02 14:20:26.041861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.041818 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-d9c3cc" Nov 02 14:20:26.085030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.082268 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f590a48c-0e64-4bca-8401-fcacb8386fce-kube-api-access-wspsf\" (UniqueName: \"kubernetes.io/projected/f590a48c-0e64-4bca-8401-fcacb8386fce-kube-api-access-wspsf\") pod \"f590a48c-0e64-4bca-8401-fcacb8386fce\" (UID: \"f590a48c-0e64-4bca-8401-fcacb8386fce\") " Nov 02 14:20:26.090019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.089936 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f590a48c-0e64-4bca-8401-fcacb8386fce-kube-api-access-wspsf" pod "f590a48c-0e64-4bca-8401-fcacb8386fce" (UID: "f590a48c-0e64-4bca-8401-fcacb8386fce"). InnerVolumeSpecName "kube-api-access-wspsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:26.092468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.092428 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wspsf\" (UniqueName: \"kubernetes.io/projected/f590a48c-0e64-4bca-8401-fcacb8386fce-kube-api-access-wspsf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:26.340283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.340196 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:26.346228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.346200 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:26.890120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.890081 5591 generic.go:356] "Generic (PLEG): container finished" podID="f590a48c-0e64-4bca-8401-fcacb8386fce" containerID="92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd" exitCode=137 Nov 02 14:20:26.890296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.890125 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7039-d9c3cc" Nov 02 14:20:26.890296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.890188 5591 scope.go:122] "RemoveContainer" containerID="92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd" Nov 02 14:20:26.890610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.890579 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:26.891813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.891792 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:20:26.900054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.900030 5591 scope.go:122] "RemoveContainer" containerID="92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd" Nov 02 14:20:26.900352 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:26.900329 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd\": container with ID starting with 92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd not found: ID does not exist" containerID="92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd" Nov 02 14:20:26.900432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:26.900365 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd"} err="failed to get container status \"92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd\": rpc error: code = NotFound desc = could not find container \"92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd\": container with ID starting with 92d601352ec6aa10e663a2fa00a65455e8881fe7aa1f8954055319cc3e41a4fd not found: ID does not exist" Nov 02 14:20:26.965477 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:26.965438 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7039-d9c3cc\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="f590a48c-0e64-4bca-8401-fcacb8386fce" pod="opendatahub/test-curl-7039-d9c3cc" Nov 02 14:20:27.103855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:27.102485 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:27.431294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:27.431257 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f590a48c-0e64-4bca-8401-fcacb8386fce" path="/var/lib/kubelet/pods/f590a48c-0e64-4bca-8401-fcacb8386fce/volumes" Nov 02 14:20:28.407635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:28.407596 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:28.577633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:28.577602 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:29.034926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:29.032837 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:30.055999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.055953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:30.222087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.222044 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:30.304047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.298747 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7"] Nov 02 14:20:30.336153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.336054 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7"] Nov 02 14:20:30.336512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.336476 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:30.360951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.360898 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbq8w\" (UniqueName: \"kubernetes.io/projected/13100993-e396-430e-9692-694b284426fc-kube-api-access-xbq8w\") pod \"payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7\" (UID: \"13100993-e396-430e-9692-694b284426fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:30.361255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.361106 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13100993-e396-430e-9692-694b284426fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7\" (UID: \"13100993-e396-430e-9692-694b284426fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:30.398013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.392408 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-26b508\"" Nov 02 14:20:30.398013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.392488 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-26b508-dockercfg-8t2cj\"" Nov 02 14:20:30.462145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.462100 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13100993-e396-430e-9692-694b284426fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7\" (UID: \"13100993-e396-430e-9692-694b284426fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:30.462340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.462228 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbq8w\" (UniqueName: \"kubernetes.io/projected/13100993-e396-430e-9692-694b284426fc-kube-api-access-xbq8w\") pod \"payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7\" (UID: \"13100993-e396-430e-9692-694b284426fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:30.463357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.463324 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13100993-e396-430e-9692-694b284426fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7\" (UID: \"13100993-e396-430e-9692-694b284426fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:30.486887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.486855 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbq8w\" (UniqueName: \"kubernetes.io/projected/13100993-e396-430e-9692-694b284426fc-kube-api-access-xbq8w\") pod \"payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7\" (UID: \"13100993-e396-430e-9692-694b284426fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:30.733105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.733075 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:30.784006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:30.779889 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:31.075350 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:31.075249 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:20:31.078829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:31.078803 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7"] Nov 02 14:20:31.080309 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:31.080276 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13100993_e396_430e_9692_694b284426fc.slice/crio-8df1e27e67217df30099fef1dda032b38363d865958adcfa3f9059639adf49ae WatchSource:0}: Error finding container 8df1e27e67217df30099fef1dda032b38363d865958adcfa3f9059639adf49ae: Status 404 returned error can't find the container with id 8df1e27e67217df30099fef1dda032b38363d865958adcfa3f9059639adf49ae Nov 02 14:20:31.927322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:31.927267 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" event={"ID":"13100993-e396-430e-9692-694b284426fc","Type":"ContainerStarted","Data":"59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512"} Nov 02 14:20:31.927487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:31.927335 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" event={"ID":"13100993-e396-430e-9692-694b284426fc","Type":"ContainerStarted","Data":"8df1e27e67217df30099fef1dda032b38363d865958adcfa3f9059639adf49ae"} Nov 02 14:20:31.927487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:31.927371 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:31.971409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:31.971356 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" podStartSLOduration=1.9713433729999998 podStartE2EDuration="1.971343373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:31.968661628 +0000 UTC m=+2180.694283398" watchObservedRunningTime="2026-11-02 14:20:31.971343373 +0000 UTC m=+2180.696965144" Nov 02 14:20:32.173352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:32.173314 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:32.339187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:32.339105 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:32.499018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:32.498955 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:34.088503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:34.088460 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:34.664007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:34.663957 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:36.011189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:36.011108 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:36.189626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:36.189591 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:36.700009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:36.686780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:36.871182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:36.871146 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:38.003675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:38.003639 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:39.410555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:39.410514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:39.763849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:39.763755 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:39.928364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:39.928329 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:40.501657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:40.501611 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:40.959357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:40.959314 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:41.751651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:41.751609 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:42.937281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:42.937250 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:20:43.242838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.242758 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7030-3e1b45"] Nov 02 14:20:43.246899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.246878 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-3e1b45" Nov 02 14:20:43.271023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.262873 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7jzh4\"" Nov 02 14:20:43.271023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.263564 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7030-3e1b45"] Nov 02 14:20:43.279815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.279781 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnpg\" (UniqueName: \"kubernetes.io/projected/750e0753-370a-42ac-a865-f3ad5664fa97-kube-api-access-kcnpg\") pod \"test-curl-7030-3e1b45\" (UID: \"750e0753-370a-42ac-a865-f3ad5664fa97\") " pod="opendatahub/test-curl-7030-3e1b45" Nov 02 14:20:43.380841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.380808 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnpg\" (UniqueName: \"kubernetes.io/projected/750e0753-370a-42ac-a865-f3ad5664fa97-kube-api-access-kcnpg\") pod \"test-curl-7030-3e1b45\" (UID: \"750e0753-370a-42ac-a865-f3ad5664fa97\") " pod="opendatahub/test-curl-7030-3e1b45" Nov 02 14:20:43.407715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.407689 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnpg\" (UniqueName: \"kubernetes.io/projected/750e0753-370a-42ac-a865-f3ad5664fa97-kube-api-access-kcnpg\") pod \"test-curl-7030-3e1b45\" (UID: \"750e0753-370a-42ac-a865-f3ad5664fa97\") " pod="opendatahub/test-curl-7030-3e1b45" Nov 02 14:20:43.462478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.462438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:43.560268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.560186 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-3e1b45" Nov 02 14:20:43.865866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:43.865836 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7030-3e1b45"] Nov 02 14:20:43.867270 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:43.867240 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod750e0753_370a_42ac_a865_f3ad5664fa97.slice/crio-057f67111524fc7ef1d8dfee3a08fe2aaedf180ab1462aeb7a3f53af743c574b WatchSource:0}: Error finding container 057f67111524fc7ef1d8dfee3a08fe2aaedf180ab1462aeb7a3f53af743c574b: Status 404 returned error can't find the container with id 057f67111524fc7ef1d8dfee3a08fe2aaedf180ab1462aeb7a3f53af743c574b Nov 02 14:20:44.041055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:44.041017 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7030-3e1b45" event={"ID":"750e0753-370a-42ac-a865-f3ad5664fa97","Type":"ContainerStarted","Data":"6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09"} Nov 02 14:20:44.041505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:44.041064 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7030-3e1b45" event={"ID":"750e0753-370a-42ac-a865-f3ad5664fa97","Type":"ContainerStarted","Data":"057f67111524fc7ef1d8dfee3a08fe2aaedf180ab1462aeb7a3f53af743c574b"} Nov 02 14:20:44.082807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:44.082742 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7030-3e1b45" podStartSLOduration=1.082729645 podStartE2EDuration="1.082729645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:44.080434713 +0000 UTC m=+2192.806056483" watchObservedRunningTime="2026-11-02 14:20:44.082729645 +0000 UTC m=+2192.808351416" Nov 02 14:20:44.757521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:44.757485 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:44.922437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:44.922406 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:45.001880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:45.001842 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7030-3e1b45"] Nov 02 14:20:45.004719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:45.004689 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7030-3e1b45"] Nov 02 14:20:45.377015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:45.376958 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:46.055664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.055604 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7030-3e1b45" podUID="750e0753-370a-42ac-a865-f3ad5664fa97" containerName="test-curl-7030-3e1b45" containerID="cri-o://6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09" gracePeriod=2 Nov 02 14:20:46.439251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.439201 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:46.520229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.520192 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp"] Nov 02 14:20:46.520789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.520775 5591 cpu_manager.go:414] "containerMap: removing container" podUID="750e0753-370a-42ac-a865-f3ad5664fa97" containerName="test-curl-7030-3e1b45" Nov 02 14:20:46.520834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.520790 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750e0753-370a-42ac-a865-f3ad5664fa97" containerName="test-curl-7030-3e1b45" Nov 02 14:20:46.520934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.520900 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="750e0753-370a-42ac-a865-f3ad5664fa97" containerName="test-curl-7030-3e1b45" Nov 02 14:20:46.524855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.524835 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:46.542865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.542842 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp"] Nov 02 14:20:46.552543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.552505 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-662e3e\"" Nov 02 14:20:46.620194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.620162 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:46.629171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.629133 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp\" (UID: \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:46.629388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.629369 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69n55\" (UniqueName: \"kubernetes.io/projected/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-kube-api-access-69n55\") pod \"payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp\" (UID: \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:46.738067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.737937 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp\" (UID: \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:46.738254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.738185 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69n55\" (UniqueName: \"kubernetes.io/projected/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-kube-api-access-69n55\") pod \"payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp\" (UID: \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:46.742005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.739600 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp\" (UID: \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:46.768017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.767670 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69n55\" (UniqueName: \"kubernetes.io/projected/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-kube-api-access-69n55\") pod \"payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp\" (UID: \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:46.839611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:46.839574 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:47.150713 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:47.150681 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda6f83b_4ff5_4028_ab2a_05679c2e6fb8.slice/crio-16173fb4323c57b7b93af2f97d13dffdbb879d9c8fb7cab9773c84999e9329b4 WatchSource:0}: Error finding container 16173fb4323c57b7b93af2f97d13dffdbb879d9c8fb7cab9773c84999e9329b4: Status 404 returned error can't find the container with id 16173fb4323c57b7b93af2f97d13dffdbb879d9c8fb7cab9773c84999e9329b4 Nov 02 14:20:47.151484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:47.151454 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp"] Nov 02 14:20:47.424607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:47.424509 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:47.611585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:47.611552 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2"] Nov 02 14:20:47.690907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:47.690791 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:48.073174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:48.073141 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" event={"ID":"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8","Type":"ContainerStarted","Data":"b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2"} Nov 02 14:20:48.073343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:48.073196 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" event={"ID":"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8","Type":"ContainerStarted","Data":"16173fb4323c57b7b93af2f97d13dffdbb879d9c8fb7cab9773c84999e9329b4"} Nov 02 14:20:48.073343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:48.073235 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:48.132035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:48.128582 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" podStartSLOduration=2.128563236 podStartE2EDuration="2.128563236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:48.127155769 +0000 UTC m=+2196.852777538" watchObservedRunningTime="2026-11-02 14:20:48.128563236 +0000 UTC m=+2196.854185007" Nov 02 14:20:48.339617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:48.339590 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-3e1b45" Nov 02 14:20:48.368946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:48.368907 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750e0753-370a-42ac-a865-f3ad5664fa97-kube-api-access-kcnpg\" (UniqueName: \"kubernetes.io/projected/750e0753-370a-42ac-a865-f3ad5664fa97-kube-api-access-kcnpg\") pod \"750e0753-370a-42ac-a865-f3ad5664fa97\" (UID: \"750e0753-370a-42ac-a865-f3ad5664fa97\") " Nov 02 14:20:48.371121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:48.371094 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750e0753-370a-42ac-a865-f3ad5664fa97-kube-api-access-kcnpg" pod "750e0753-370a-42ac-a865-f3ad5664fa97" (UID: "750e0753-370a-42ac-a865-f3ad5664fa97"). InnerVolumeSpecName "kube-api-access-kcnpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:48.483265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:48.478327 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcnpg\" (UniqueName: \"kubernetes.io/projected/750e0753-370a-42ac-a865-f3ad5664fa97-kube-api-access-kcnpg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:20:49.080478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:49.080436 5591 generic.go:356] "Generic (PLEG): container finished" podID="750e0753-370a-42ac-a865-f3ad5664fa97" containerID="6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09" exitCode=137 Nov 02 14:20:49.080916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:49.080499 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7030-3e1b45" Nov 02 14:20:49.080916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:49.080538 5591 scope.go:122] "RemoveContainer" containerID="6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09" Nov 02 14:20:49.090762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:49.090740 5591 scope.go:122] "RemoveContainer" containerID="6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09" Nov 02 14:20:49.091067 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:49.091043 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09\": container with ID starting with 6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09 not found: ID does not exist" containerID="6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09" Nov 02 14:20:49.091121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:49.091077 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09"} err="failed to get container status \"6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09\": rpc error: code = NotFound desc = could not find container \"6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09\": container with ID starting with 6fb9a8740784e5b53282ad5b122aa274e73a6662e9841ddad44a4c3ed7315c09 not found: ID does not exist" Nov 02 14:20:49.104595 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:20:49.104562 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7030-3e1b45\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="750e0753-370a-42ac-a865-f3ad5664fa97" pod="opendatahub/test-curl-7030-3e1b45" Nov 02 14:20:49.225494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:49.225452 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:49.433727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:49.433689 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750e0753-370a-42ac-a865-f3ad5664fa97" path="/var/lib/kubelet/pods/750e0753-370a-42ac-a865-f3ad5664fa97/volumes" Nov 02 14:20:50.788013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:50.781905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:52.656006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:52.647479 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:52.840159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:52.840116 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:53.012285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:53.012204 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:54.450092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:54.450055 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:54.619283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:54.619249 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:56.258547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.258511 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c"] Nov 02 14:20:56.264355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.264333 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.302022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.294224 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-26b508-serving-cert\"" Nov 02 14:20:56.302022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.298638 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c"] Nov 02 14:20:56.403340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.403297 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:56.407840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.407819 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-metrics-certs\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.407941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.407864 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b2c8e3-58b7-447b-b179-1e83d6543efe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.408026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.407976 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.408088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.408042 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8rx\" (UniqueName: \"kubernetes.io/projected/43b2c8e3-58b7-447b-b179-1e83d6543efe-kube-api-access-xd8rx\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.510095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.509971 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b2c8e3-58b7-447b-b179-1e83d6543efe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.510095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.510083 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.510341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.510104 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8rx\" (UniqueName: \"kubernetes.io/projected/43b2c8e3-58b7-447b-b179-1e83d6543efe-kube-api-access-xd8rx\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.510341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.510317 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-metrics-certs\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.510738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.510711 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b2c8e3-58b7-447b-b179-1e83d6543efe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.512727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.512699 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.512836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.512699 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-metrics-certs\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.534711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.534677 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8rx\" (UniqueName: \"kubernetes.io/projected/43b2c8e3-58b7-447b-b179-1e83d6543efe-kube-api-access-xd8rx\") pod \"maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.577997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.577948 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:56.580106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.580070 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:56.886457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:56.886429 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c"] Nov 02 14:20:56.887682 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:20:56.887658 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b2c8e3_58b7_447b_b179_1e83d6543efe.slice/crio-5e0be3d2be21f5a8ecaaaa9ea4c6c8e7c2a94914dc5f79fddb1dcffbf4e005ab WatchSource:0}: Error finding container 5e0be3d2be21f5a8ecaaaa9ea4c6c8e7c2a94914dc5f79fddb1dcffbf4e005ab: Status 404 returned error can't find the container with id 5e0be3d2be21f5a8ecaaaa9ea4c6c8e7c2a94914dc5f79fddb1dcffbf4e005ab Nov 02 14:20:57.134777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:57.134693 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" event={"ID":"43b2c8e3-58b7-447b-b179-1e83d6543efe","Type":"ContainerStarted","Data":"5e0be3d2be21f5a8ecaaaa9ea4c6c8e7c2a94914dc5f79fddb1dcffbf4e005ab"} Nov 02 14:20:58.141169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:58.139836 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:20:58.163593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:58.163552 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" event={"ID":"43b2c8e3-58b7-447b-b179-1e83d6543efe","Type":"ContainerStarted","Data":"c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d"} Nov 02 14:20:58.165232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:58.165204 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:20:58.230814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:58.230762 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" podStartSLOduration=1.879106792 podStartE2EDuration="2.230746794s" totalImagesPullingTime="351.640002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:58.227700621 +0000 UTC m=+2206.953322390" watchObservedRunningTime="2026-11-02 14:20:58.230746794 +0000 UTC m=+2206.956368609" Nov 02 14:20:59.083766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:59.083735 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:20:59.838336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:20:59.838299 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:00.006101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:00.006068 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:01.074944 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:21:01.074899 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:21:01.552963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:01.552925 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:02.248053 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:02.247964 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:02.416852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:02.416814 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:02.634908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:02.634864 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" podUID="8295ec02-0edd-401e-8973-44ea94e52e90" containerName="main" containerID="cri-o://c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e" gracePeriod=45 Nov 02 14:21:02.911995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:02.911955 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:21:03.060871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.060835 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8295ec02-0edd-401e-8973-44ea94e52e90-kube-api-access-jrqct\" (UniqueName: \"kubernetes.io/projected/8295ec02-0edd-401e-8973-44ea94e52e90-kube-api-access-jrqct\") pod \"8295ec02-0edd-401e-8973-44ea94e52e90\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " Nov 02 14:21:03.061113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.060929 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-home\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-home\") pod \"8295ec02-0edd-401e-8973-44ea94e52e90\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " Nov 02 14:21:03.061113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.060957 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-model-cache\") pod \"8295ec02-0edd-401e-8973-44ea94e52e90\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " Nov 02 14:21:03.061113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061057 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-dshm\") pod \"8295ec02-0edd-401e-8973-44ea94e52e90\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " Nov 02 14:21:03.061113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061093 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-tmp-dir\") pod \"8295ec02-0edd-401e-8973-44ea94e52e90\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " Nov 02 14:21:03.061369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061126 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8295ec02-0edd-401e-8973-44ea94e52e90-tls-certs\" (UniqueName: \"kubernetes.io/secret/8295ec02-0edd-401e-8973-44ea94e52e90-tls-certs\") pod \"8295ec02-0edd-401e-8973-44ea94e52e90\" (UID: \"8295ec02-0edd-401e-8973-44ea94e52e90\") " Nov 02 14:21:03.061369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061262 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-home" pod "8295ec02-0edd-401e-8973-44ea94e52e90" (UID: "8295ec02-0edd-401e-8973-44ea94e52e90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:03.061369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061301 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-model-cache" pod "8295ec02-0edd-401e-8973-44ea94e52e90" (UID: "8295ec02-0edd-401e-8973-44ea94e52e90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:03.061492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061406 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-tmp-dir" pod "8295ec02-0edd-401e-8973-44ea94e52e90" (UID: "8295ec02-0edd-401e-8973-44ea94e52e90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:03.061843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061818 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:03.061951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061848 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:03.061951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.061863 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:03.063276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.063253 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8295ec02-0edd-401e-8973-44ea94e52e90-tls-certs" pod "8295ec02-0edd-401e-8973-44ea94e52e90" (UID: "8295ec02-0edd-401e-8973-44ea94e52e90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:21:03.063398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.063378 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8295ec02-0edd-401e-8973-44ea94e52e90-kube-api-access-jrqct" pod "8295ec02-0edd-401e-8973-44ea94e52e90" (UID: "8295ec02-0edd-401e-8973-44ea94e52e90"). InnerVolumeSpecName "kube-api-access-jrqct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:03.063448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.063392 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-dshm" pod "8295ec02-0edd-401e-8973-44ea94e52e90" (UID: "8295ec02-0edd-401e-8973-44ea94e52e90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:03.074700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.074676 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f"] Nov 02 14:21:03.074971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.074923 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" containerID="cri-o://b966e1d6eda774d8c3e231bf4359741f320982bc2f1dcc29669383f1b2251454" gracePeriod=30 Nov 02 14:21:03.162761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.162682 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8295ec02-0edd-401e-8973-44ea94e52e90-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:03.162761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.162714 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrqct\" (UniqueName: \"kubernetes.io/projected/8295ec02-0edd-401e-8973-44ea94e52e90-kube-api-access-jrqct\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:03.162761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.162725 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8295ec02-0edd-401e-8973-44ea94e52e90-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:03.215851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.215801 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:03.234834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.217765 5591 generic.go:356] "Generic (PLEG): container finished" podID="8295ec02-0edd-401e-8973-44ea94e52e90" containerID="c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e" exitCode=0 Nov 02 14:21:03.234834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.218014 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" event={"ID":"8295ec02-0edd-401e-8973-44ea94e52e90","Type":"ContainerDied","Data":"c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e"} Nov 02 14:21:03.234834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.218075 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" event={"ID":"8295ec02-0edd-401e-8973-44ea94e52e90","Type":"ContainerDied","Data":"d6551022585e50ca959d1bfed363ac5d74a27996a46134931e583c032ceeb782"} Nov 02 14:21:03.234834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.218099 5591 scope.go:122] "RemoveContainer" containerID="c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e" Nov 02 14:21:03.234834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.218291 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2" Nov 02 14:21:03.255725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.255706 5591 scope.go:122] "RemoveContainer" containerID="c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e" Nov 02 14:21:03.256133 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:21:03.256096 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e\": container with ID starting with c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e not found: ID does not exist" containerID="c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e" Nov 02 14:21:03.256133 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.256126 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e"} err="failed to get container status \"c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e\": rpc error: code = NotFound desc = could not find container \"c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e\": container with ID starting with c4614be284c5bad9a86658879e5b9cca71eac57fd28052d3d04504335649183e not found: ID does not exist" Nov 02 14:21:03.293435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.293399 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2"] Nov 02 14:21:03.297588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.297564 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4b7759/auth-test-model-4b7759-kserve-5c85c8b7b5-mwrr2"] Nov 02 14:21:03.431171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:03.431086 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8295ec02-0edd-401e-8973-44ea94e52e90" path="/var/lib/kubelet/pods/8295ec02-0edd-401e-8973-44ea94e52e90/volumes" Nov 02 14:21:04.773005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:04.772348 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:05.183364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:05.183334 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:21:06.373545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:06.373473 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:08.063001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.062922 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:08.255642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.255597 5591 generic.go:356] "Generic (PLEG): container finished" podID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerID="b966e1d6eda774d8c3e231bf4359741f320982bc2f1dcc29669383f1b2251454" exitCode=0 Nov 02 14:21:08.255808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.255673 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" event={"ID":"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4","Type":"ContainerDied","Data":"b966e1d6eda774d8c3e231bf4359741f320982bc2f1dcc29669383f1b2251454"} Nov 02 14:21:08.345421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.345394 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:21:08.441722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.441677 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-certs\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.441919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.441801 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-podinfo\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.441919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.441845 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-token\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-token\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.441919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.441898 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-kube-api-access-j9hj7\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-kube-api-access-j9hj7\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.442165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442060 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-credential-socket\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.442165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442092 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-envoy\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.442282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442179 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-socket\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.442282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442224 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-data\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.442282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442263 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istiod-ca-cert\") pod \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\" (UID: \"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4\") " Nov 02 14:21:08.442448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442387 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-certs" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:08.442609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442588 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-data" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:08.442829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442800 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-socket" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:08.442925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442812 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:08.442925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442860 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:08.442925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442859 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istiod-ca-cert" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:08.443131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.442927 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-credential-socket" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:08.444635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.444611 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-podinfo" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:21:08.444758 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.444667 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-token" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:08.444889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.444870 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-kube-api-access-j9hj7" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "kube-api-access-j9hj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:08.445021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.444951 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-envoy" pod "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" (UID: "deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:08.544383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.544343 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:08.544579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.544398 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:08.544579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.544439 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:08.544579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.544456 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:08.544579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.544475 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:08.544579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.544491 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:08.544579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:08.544508 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9hj7\" (UniqueName: \"kubernetes.io/projected/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4-kube-api-access-j9hj7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:09.264555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:09.264520 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" Nov 02 14:21:09.264957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:09.264521 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f" event={"ID":"deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4","Type":"ContainerDied","Data":"36f719ca6c59f41208f66512fc889cd4d912a421d3daa22614176100a03bc0be"} Nov 02 14:21:09.264957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:09.264694 5591 scope.go:122] "RemoveContainer" containerID="b966e1d6eda774d8c3e231bf4359741f320982bc2f1dcc29669383f1b2251454" Nov 02 14:21:09.342478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:09.342440 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f"] Nov 02 14:21:09.355386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:09.355352 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-htc6f"] Nov 02 14:21:09.430245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:09.430211 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" path="/var/lib/kubelet/pods/deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4/volumes" Nov 02 14:21:09.756261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:09.756208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:09.885816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:09.885782 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:21:11.446875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.446818 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:11.616193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.616156 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:11.790773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.790674 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms"] Nov 02 14:21:11.792247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.792220 5591 cpu_manager.go:414] "containerMap: removing container" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" Nov 02 14:21:11.792489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.792473 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" Nov 02 14:21:11.792873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.792757 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8295ec02-0edd-401e-8973-44ea94e52e90" containerName="main" Nov 02 14:21:11.793892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.793873 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8295ec02-0edd-401e-8973-44ea94e52e90" containerName="main" Nov 02 14:21:11.794199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.794180 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8295ec02-0edd-401e-8973-44ea94e52e90" containerName="main" Nov 02 14:21:11.794340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.794326 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="deaf7bd7-7b6d-4660-9d46-b10a2a0ea8c4" containerName="istio-proxy" Nov 02 14:21:11.822301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.822263 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms"] Nov 02 14:21:11.822528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.822377 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.840877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.840842 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-87efd3-openshift-default-dockercfg-l9rbz\"" Nov 02 14:21:11.883861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.883820 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:11.964008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.963831 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-socket\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.964008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.963946 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-envoy\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.964008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.964011 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-credential-socket\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.964345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.964058 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8845a851-daf1-4d64-9341-ea6b4d256ce5-istiod-ca-cert\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.964345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.964122 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-token\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.964345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.964185 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-data\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.964345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.964229 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-certs\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.964345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.964262 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-podinfo\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:11.964643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:11.964382 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-kube-api-access-vb25b\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.065799 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-kube-api-access-vb25b\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.065879 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-socket\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.065967 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-envoy\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.066028 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-credential-socket\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.066069 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8845a851-daf1-4d64-9341-ea6b4d256ce5-istiod-ca-cert\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.066129 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-token\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.066185 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-data\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.066224 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-certs\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.066253 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-podinfo\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.068263 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8845a851-daf1-4d64-9341-ea6b4d256ce5-istiod-ca-cert\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.068586 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-data\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.068816 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-socket\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.081057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.069399 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-certs\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.093702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.083918 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-credential-socket\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.093702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.084374 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:12.126111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.114407 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-podinfo\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.127567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.127102 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-token\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.127567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.127534 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-envoy\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.130073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.130016 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-kube-api-access-vb25b\") pod \"e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.156634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.156333 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:12.493664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.493632 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms"] Nov 02 14:21:12.494324 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:21:12.494294 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8845a851_daf1_4d64_9341_ea6b4d256ce5.slice/crio-6a8a3c5aec8e30bdf45ef01ccde4682f9eac66217bd04ba2f9c96525a274b011 WatchSource:0}: Error finding container 6a8a3c5aec8e30bdf45ef01ccde4682f9eac66217bd04ba2f9c96525a274b011: Status 404 returned error can't find the container with id 6a8a3c5aec8e30bdf45ef01ccde4682f9eac66217bd04ba2f9c96525a274b011 Nov 02 14:21:12.497353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.497315 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:12.497465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.497399 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:12.497465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.497447 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:12.762129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:12.756914 5591 scope.go:122] "RemoveContainer" containerID="7a1d8032bf695c872b3fa22d44d456cf9c06ff0122e6f0be4f72f82e467f8baa" Nov 02 14:21:13.077634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.077556 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6"] Nov 02 14:21:13.083910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.083888 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.111069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.111038 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-662e3e-serving-cert\"" Nov 02 14:21:13.115532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.115496 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6"] Nov 02 14:21:13.184103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.184068 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c30329ab-6930-4243-826e-4f1733f6172a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.184295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.184232 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.184353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.184336 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vs56\" (UniqueName: \"kubernetes.io/projected/c30329ab-6930-4243-826e-4f1733f6172a-kube-api-access-8vs56\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.184437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.184426 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.227826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.227793 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:13.288007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.287817 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.288007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.287912 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c30329ab-6930-4243-826e-4f1733f6172a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.288286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.288065 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.288286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.288162 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vs56\" (UniqueName: \"kubernetes.io/projected/c30329ab-6930-4243-826e-4f1733f6172a-kube-api-access-8vs56\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.294005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.289917 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c30329ab-6930-4243-826e-4f1733f6172a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.313014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.311493 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" event={"ID":"8845a851-daf1-4d64-9341-ea6b4d256ce5","Type":"ContainerStarted","Data":"6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca"} Nov 02 14:21:13.313014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.311568 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" event={"ID":"8845a851-daf1-4d64-9341-ea6b4d256ce5","Type":"ContainerStarted","Data":"6a8a3c5aec8e30bdf45ef01ccde4682f9eac66217bd04ba2f9c96525a274b011"} Nov 02 14:21:13.329675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.324082 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vs56\" (UniqueName: \"kubernetes.io/projected/c30329ab-6930-4243-826e-4f1733f6172a-kube-api-access-8vs56\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.329675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.328524 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.330731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.330520 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.377169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.377099 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" podStartSLOduration=2.377085231 podStartE2EDuration="2.377085231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:13.374485149 +0000 UTC m=+2222.100106913" watchObservedRunningTime="2026-11-02 14:21:13.377085231 +0000 UTC m=+2222.102707001" Nov 02 14:21:13.398246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.398211 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:13.415349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.415311 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:13.612272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.612190 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:13.781550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:13.781518 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6"] Nov 02 14:21:13.782176 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:21:13.782146 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30329ab_6930_4243_826e_4f1733f6172a.slice/crio-091761f680f03ea77b828063b3afceb9aece209a4393f583eb3a3ad8415e4378 WatchSource:0}: Error finding container 091761f680f03ea77b828063b3afceb9aece209a4393f583eb3a3ad8415e4378: Status 404 returned error can't find the container with id 091761f680f03ea77b828063b3afceb9aece209a4393f583eb3a3ad8415e4378 Nov 02 14:21:14.158327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:14.158292 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:14.164524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:14.164496 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:14.319927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:14.319889 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" event={"ID":"c30329ab-6930-4243-826e-4f1733f6172a","Type":"ContainerStarted","Data":"f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7"} Nov 02 14:21:14.320144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:14.320024 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:14.320208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:14.320142 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:14.320208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:14.320173 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" event={"ID":"c30329ab-6930-4243-826e-4f1733f6172a","Type":"ContainerStarted","Data":"091761f680f03ea77b828063b3afceb9aece209a4393f583eb3a3ad8415e4378"} Nov 02 14:21:14.320958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:14.320939 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:21:14.487066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:14.470612 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" podStartSLOduration=1.079284343 podStartE2EDuration="1.470587464s" totalImagesPullingTime="391.303121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:14.398402705 +0000 UTC m=+2223.124024475" watchObservedRunningTime="2026-11-02 14:21:14.470587464 +0000 UTC m=+2223.196209232" Nov 02 14:21:15.257697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:15.257613 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:15.919357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:15.919324 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:16.109309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:16.109239 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:17.411208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:17.411087 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:17.696765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:17.696675 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:17.865120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:17.865070 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:19.131336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.131289 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:19.759249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.759210 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk"] Nov 02 14:21:19.763620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.763596 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.780368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.780343 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-0d1ca7-openshift-default-dockercfg-9n4nr\"" Nov 02 14:21:19.798822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.798783 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk"] Nov 02 14:21:19.834881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.834841 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.835088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.835006 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1537b9-afe7-43be-81c0-0762bd50a35f-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.835146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.835095 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.835197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.835137 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.835235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.835200 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.835279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.835267 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.835492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.835464 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.835533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.835512 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np65x\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-kube-api-access-np65x\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.835590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.835573 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.936859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.936824 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.936882 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.936931 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1537b9-afe7-43be-81c0-0762bd50a35f-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.936997 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937024 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937065 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937154 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937269 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937296 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np65x\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-kube-api-access-np65x\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937387 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937572 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937711 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937763 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.937932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.937909 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1537b9-afe7-43be-81c0-0762bd50a35f-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.940318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.940296 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.940482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.940465 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.962210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.962175 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:19.962342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:19.962183 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np65x\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-kube-api-access-np65x\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:20.080047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:20.079955 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:20.333016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:20.332923 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:21:20.436219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:20.436187 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk"] Nov 02 14:21:20.439456 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:21:20.439425 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1537b9_afe7_43be_81c0_0762bd50a35f.slice/crio-f0f0d7e6031d5832fe3fd01b6734aa9adf1e491e12bf05de3fde60fbc8d711b8 WatchSource:0}: Error finding container f0f0d7e6031d5832fe3fd01b6734aa9adf1e491e12bf05de3fde60fbc8d711b8: Status 404 returned error can't find the container with id f0f0d7e6031d5832fe3fd01b6734aa9adf1e491e12bf05de3fde60fbc8d711b8 Nov 02 14:21:20.442341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:20.442310 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:20.442437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:20.442374 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:20.442437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:20.442406 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:20.678959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:20.673061 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:20.842016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:20.840498 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:21.380900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:21.380853 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" event={"ID":"4c1537b9-afe7-43be-81c0-0762bd50a35f","Type":"ContainerStarted","Data":"3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042"} Nov 02 14:21:21.383490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:21.381011 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" event={"ID":"4c1537b9-afe7-43be-81c0-0762bd50a35f","Type":"ContainerStarted","Data":"f0f0d7e6031d5832fe3fd01b6734aa9adf1e491e12bf05de3fde60fbc8d711b8"} Nov 02 14:21:21.467328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:21.467241 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podStartSLOduration=2.467220143 podStartE2EDuration="2.467220143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:21.442270427 +0000 UTC m=+2230.167892197" watchObservedRunningTime="2026-11-02 14:21:21.467220143 +0000 UTC m=+2230.192841914" Nov 02 14:21:22.080365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.080328 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:22.081776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.081748 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:21:22.081910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.081797 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:21:22.389600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.389561 5591 generic.go:356] "Generic (PLEG): container finished" podID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerID="6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407" exitCode=0 Nov 02 14:21:22.390102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.389608 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" event={"ID":"c39babc9-1aa2-4878-8b21-b98ddd3a7f34","Type":"ContainerDied","Data":"6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407"} Nov 02 14:21:22.390342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.390327 5591 scope.go:122] "RemoveContainer" containerID="6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407" Nov 02 14:21:22.392026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.391966 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:22.392132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.392069 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:22.392132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.392103 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:22.426524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.426494 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:22.602638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:22.602595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:23.081103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:23.081067 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:21:23.081324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:23.081116 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:21:23.411646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:23.411604 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" event={"ID":"c39babc9-1aa2-4878-8b21-b98ddd3a7f34","Type":"ContainerStarted","Data":"4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09"} Nov 02 14:21:24.082040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:24.081382 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:21:24.082040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:24.081445 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:21:24.269007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:24.268909 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:24.349641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:24.344481 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:21:24.360148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:24.353155 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 14:21:24.360148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:24.353258 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 14:21:24.512624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:24.512590 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:25.080598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:25.080548 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:21:25.080817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:25.080632 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:21:25.343174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:25.343083 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 14:21:25.343443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:25.343411 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 14:21:25.919002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:25.912403 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:26.083006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:26.082423 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:21:26.083006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:26.082502 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:21:26.348068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:26.342384 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 14:21:26.348068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:26.342451 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 14:21:26.891274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:26.891238 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:21:27.081149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:27.081106 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:21:27.081637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:27.081171 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:21:27.343547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:27.343447 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 14:21:27.343830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:27.343796 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 14:21:27.584007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:27.582659 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:28.085925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:28.085888 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:28.086406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:28.086395 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:28.087066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:28.087041 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:21:28.356223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:28.356111 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 14:21:28.356223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:28.356188 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 14:21:28.943130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:28.943091 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4"] Nov 02 14:21:29.037305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:29.037266 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:29.380431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:29.380350 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:21:29.381466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:29.381446 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:21:29.506070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:29.506021 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:29.786532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:29.786498 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:30.344272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:30.344235 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:30.584927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:30.584890 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:31.074715 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:21:31.074680 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:21:31.573007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:31.567586 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:32.832764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:32.832728 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:33.518008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:33.513875 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:34.439461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:34.439424 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:34.658940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:34.658905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:35.358632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:35.358592 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:36.072780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:36.072740 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:36.241943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:36.241905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:37.208006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:37.207414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:38.873121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:38.873076 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:39.615742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:39.615672 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:40.094753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:40.094713 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:21:40.569487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:40.569449 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:41.093198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.093157 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:41.416753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.412822 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb"] Nov 02 14:21:41.431823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.431790 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.454621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.454589 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb"] Nov 02 14:21:41.459282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.459253 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ee3eca-openshift-default-dockercfg-4qcl9\"" Nov 02 14:21:41.513810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.513768 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-credential-socket\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.514092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.513824 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-envoy\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.514092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.513937 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98702400-9887-4120-adea-809091fdf0aa-istiod-ca-cert\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.514092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.514021 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98702400-9887-4120-adea-809091fdf0aa-istio-podinfo\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.514092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.514080 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-data\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.514341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.514172 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-socket\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.514341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.514233 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-istio-token\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.514341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.514309 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-certs\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.514508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.514361 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7r6\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-kube-api-access-6h7r6\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.547524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.547474 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:41.619881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.619839 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-envoy\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.619937 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98702400-9887-4120-adea-809091fdf0aa-istiod-ca-cert\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.620022 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98702400-9887-4120-adea-809091fdf0aa-istio-podinfo\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.620083 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-data\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.620114 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-socket\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.620182 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-istio-token\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.620256 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-certs\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.620303 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7r6\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-kube-api-access-6h7r6\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.620405 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-credential-socket\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.620907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.620882 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-credential-socket\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.654015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.635383 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-socket\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.654015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.636088 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98702400-9887-4120-adea-809091fdf0aa-istiod-ca-cert\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.654015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.640332 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-certs\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.654015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.640851 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-data\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.666007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.662480 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-envoy\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.676049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.675368 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98702400-9887-4120-adea-809091fdf0aa-istio-podinfo\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.700007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.694096 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7r6\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-kube-api-access-6h7r6\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.724010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.718915 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-istio-token\") pod \"e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:41.767005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.763634 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:41.767005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:41.764147 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:42.120348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.120319 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb"] Nov 02 14:21:42.121526 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:21:42.121500 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98702400_9887_4120_adea_809091fdf0aa.slice/crio-092cbabf3685f3c788be9d64b2fb69ddf835182211172662d93ebf4d3bc043ef WatchSource:0}: Error finding container 092cbabf3685f3c788be9d64b2fb69ddf835182211172662d93ebf4d3bc043ef: Status 404 returned error can't find the container with id 092cbabf3685f3c788be9d64b2fb69ddf835182211172662d93ebf4d3bc043ef Nov 02 14:21:42.124334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.124299 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:42.124442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.124375 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:42.124442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.124407 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:42.682961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.682925 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" event={"ID":"98702400-9887-4120-adea-809091fdf0aa","Type":"ContainerStarted","Data":"27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f"} Nov 02 14:21:42.683431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.683008 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" event={"ID":"98702400-9887-4120-adea-809091fdf0aa","Type":"ContainerStarted","Data":"092cbabf3685f3c788be9d64b2fb69ddf835182211172662d93ebf4d3bc043ef"} Nov 02 14:21:42.739608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.739549 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" podStartSLOduration=1.739530232 podStartE2EDuration="1.739530232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:42.735411775 +0000 UTC m=+2251.461033545" watchObservedRunningTime="2026-11-02 14:21:42.739530232 +0000 UTC m=+2251.465152053" Nov 02 14:21:42.766108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.766070 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:42.808492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:42.808445 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:43.772477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:43.772444 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:43.965217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:43.965174 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" podUID="03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" containerName="main" containerID="cri-o://fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031" gracePeriod=45 Nov 02 14:21:44.244867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.244841 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:21:44.259110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.259074 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7"] Nov 02 14:21:44.260110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.260087 5591 cpu_manager.go:414] "containerMap: removing container" podUID="03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" containerName="main" Nov 02 14:21:44.260179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.260112 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" containerName="main" Nov 02 14:21:44.260347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.260335 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" containerName="main" Nov 02 14:21:44.264120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.264099 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.280051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.280027 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-21baa7-openshift-default-dockercfg-wj446\"" Nov 02 14:21:44.287563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.287536 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7"] Nov 02 14:21:44.362163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362073 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-dshm\") pod \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " Nov 02 14:21:44.362349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362164 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tls-certs\" (UniqueName: \"kubernetes.io/secret/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tls-certs\") pod \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " Nov 02 14:21:44.362349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362255 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-model-cache\") pod \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " Nov 02 14:21:44.362349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362300 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tmp-dir\") pod \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " Nov 02 14:21:44.362349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362342 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-kube-api-access-96j8x\" (UniqueName: \"kubernetes.io/projected/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-kube-api-access-96j8x\") pod \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " Nov 02 14:21:44.362582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362398 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-home\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-home\") pod \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\" (UID: \"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33\") " Nov 02 14:21:44.362582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362545 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-model-cache" pod "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" (UID: "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:44.362582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362558 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tmp-dir" pod "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" (UID: "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:44.362742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362662 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-credential-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.362802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362769 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-home" pod "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" (UID: "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:44.362802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362782 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-token\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.362916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362893 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f16efd-16c6-491a-91ff-2e42183de3fe-istiod-ca-cert\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.362957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.362932 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-certs\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.363084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.363063 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-envoy\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.363160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.363102 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.363160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.363135 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv7vz\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-kube-api-access-xv7vz\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.363330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.363268 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-podinfo\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.363430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.363405 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-data\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.363605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.363588 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:44.363660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.363609 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:44.363660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.363617 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:44.364566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.364547 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tls-certs" pod "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" (UID: "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:21:44.364721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.364687 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-dshm" pod "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" (UID: "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:21:44.364823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.364723 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-kube-api-access-96j8x" pod "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" (UID: "03bf3b4b-34d9-4c38-a24b-3f00f22c1e33"). InnerVolumeSpecName "kube-api-access-96j8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:44.465402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465355 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-credential-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.465609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465452 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-token\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.465609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465542 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f16efd-16c6-491a-91ff-2e42183de3fe-istiod-ca-cert\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.465609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465580 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-certs\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.465777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465646 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-envoy\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.465777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465677 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.465777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465714 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv7vz\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-kube-api-access-xv7vz\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.465777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465772 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-podinfo\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.466023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.465846 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-data\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.466077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.466028 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:44.466077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.466054 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:44.466077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.466069 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96j8x\" (UniqueName: \"kubernetes.io/projected/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33-kube-api-access-96j8x\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:21:44.467333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.466491 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-data\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.467333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.466755 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-credential-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.467814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.467787 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f16efd-16c6-491a-91ff-2e42183de3fe-istiod-ca-cert\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.468108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.468083 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-certs\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.470802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.470770 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-envoy\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.471111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.471086 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.474204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.474179 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-podinfo\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.490823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.490786 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-token\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.500469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.500435 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv7vz\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-kube-api-access-xv7vz\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.577700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.577663 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:44.699859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.699816 5591 generic.go:356] "Generic (PLEG): container finished" podID="03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" containerID="fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031" exitCode=0 Nov 02 14:21:44.700064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.699898 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" Nov 02 14:21:44.700064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.699912 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" event={"ID":"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33","Type":"ContainerDied","Data":"fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031"} Nov 02 14:21:44.700064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.699965 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4" event={"ID":"03bf3b4b-34d9-4c38-a24b-3f00f22c1e33","Type":"ContainerDied","Data":"533efda8142113022457d67e2163ed34ddf7bf6889a330c12d17c789a9ab23c9"} Nov 02 14:21:44.700064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.700026 5591 scope.go:122] "RemoveContainer" containerID="fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031" Nov 02 14:21:44.700889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.700866 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:44.705716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.705697 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:21:44.712782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.712765 5591 scope.go:122] "RemoveContainer" containerID="fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031" Nov 02 14:21:44.713113 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:21:44.713095 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031\": container with ID starting with fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031 not found: ID does not exist" containerID="fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031" Nov 02 14:21:44.713172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.713122 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031"} err="failed to get container status \"fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031\": rpc error: code = NotFound desc = could not find container \"fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031\": container with ID starting with fc783083111e3d1cbcbdad6594ad1b134a45db75aee89d94bd1286e742b80031 not found: ID does not exist" Nov 02 14:21:44.775233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.775194 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:44.778536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.778504 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4"] Nov 02 14:21:44.781750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.781722 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/auth-test-model-0d1ca7-kserve-55fccfc49c-kpgf4"] Nov 02 14:21:44.908941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.908910 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7"] Nov 02 14:21:44.911188 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:21:44.911161 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f16efd_16c6_491a_91ff_2e42183de3fe.slice/crio-5e1dadcf3180446b8a6a14dce6e41b8c1d0f2ab9d37b6c9b4933e47cdf659d75 WatchSource:0}: Error finding container 5e1dadcf3180446b8a6a14dce6e41b8c1d0f2ab9d37b6c9b4933e47cdf659d75: Status 404 returned error can't find the container with id 5e1dadcf3180446b8a6a14dce6e41b8c1d0f2ab9d37b6c9b4933e47cdf659d75 Nov 02 14:21:44.914108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.914072 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:44.914213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.914158 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:44.914213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:44.914205 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:21:45.299670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:45.299574 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:45.434535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:45.434503 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bf3b4b-34d9-4c38-a24b-3f00f22c1e33" path="/var/lib/kubelet/pods/03bf3b4b-34d9-4c38-a24b-3f00f22c1e33/volumes" Nov 02 14:21:45.714031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:45.713725 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:45.718300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:45.717818 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" event={"ID":"72f16efd-16c6-491a-91ff-2e42183de3fe","Type":"ContainerStarted","Data":"338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48"} Nov 02 14:21:45.718300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:45.717883 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" event={"ID":"72f16efd-16c6-491a-91ff-2e42183de3fe","Type":"ContainerStarted","Data":"5e1dadcf3180446b8a6a14dce6e41b8c1d0f2ab9d37b6c9b4933e47cdf659d75"} Nov 02 14:21:45.817857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:45.817801 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" podStartSLOduration=1.817787976 podStartE2EDuration="1.817787976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:45.804630133 +0000 UTC m=+2254.530251905" watchObservedRunningTime="2026-11-02 14:21:45.817787976 +0000 UTC m=+2254.543409746" Nov 02 14:21:46.578411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:46.578366 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:46.580247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:46.580222 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 14:21:46.580355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:46.580272 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 14:21:46.974305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:46.974264 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:47.162086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:47.153387 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:47.345067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:47.344678 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:47.578054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:47.578019 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 14:21:47.578209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:47.578074 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 14:21:48.578703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:48.578661 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 14:21:48.579161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:48.578726 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 14:21:48.921261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:48.921226 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:49.081384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:49.081324 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:49.249068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:49.248515 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:49.579094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:49.578973 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 14:21:49.579094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:49.579067 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 14:21:50.586009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:50.584488 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 02 14:21:50.586009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:50.584552 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 02 14:21:50.887402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:50.885640 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:51.241308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:51.237567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:51.583211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:51.583123 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:51.583614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:51.583600 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:51.584539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:51.584519 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:21:52.471122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:52.471074 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:53.623307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:53.623260 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:54.315995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:54.315929 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:54.505194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:54.505146 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:56.156954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:56.156906 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:58.063626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:58.063594 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:58.241007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:58.239734 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:59.802007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:59.801729 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:21:59.912800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:59.912760 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2"] Nov 02 14:21:59.918995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:59.918963 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:21:59.939396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:59.939367 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2"] Nov 02 14:21:59.949303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:59.949269 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2d13f4\"" Nov 02 14:21:59.949303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:21:59.949272 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2d13f4-dockercfg-lg4dd\"" Nov 02 14:22:00.011610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.011560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:00.067546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.067451 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a879895-0675-4c59-a0f4-ab94b9846d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2\" (UID: \"7a879895-0675-4c59-a0f4-ab94b9846d47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:00.087141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.082165 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m2d7\" (UniqueName: \"kubernetes.io/projected/7a879895-0675-4c59-a0f4-ab94b9846d47-kube-api-access-8m2d7\") pod \"payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2\" (UID: \"7a879895-0675-4c59-a0f4-ab94b9846d47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:00.189596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.189374 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a879895-0675-4c59-a0f4-ab94b9846d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2\" (UID: \"7a879895-0675-4c59-a0f4-ab94b9846d47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:00.206705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.206671 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m2d7\" (UniqueName: \"kubernetes.io/projected/7a879895-0675-4c59-a0f4-ab94b9846d47-kube-api-access-8m2d7\") pod \"payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2\" (UID: \"7a879895-0675-4c59-a0f4-ab94b9846d47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:00.208704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.199598 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:00.209572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.199191 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a879895-0675-4c59-a0f4-ab94b9846d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2\" (UID: \"7a879895-0675-4c59-a0f4-ab94b9846d47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:00.244415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.244248 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m2d7\" (UniqueName: \"kubernetes.io/projected/7a879895-0675-4c59-a0f4-ab94b9846d47-kube-api-access-8m2d7\") pod \"payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2\" (UID: \"7a879895-0675-4c59-a0f4-ab94b9846d47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:00.534287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.534249 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:00.848216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.848184 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2"] Nov 02 14:22:00.848912 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:22:00.848867 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a879895_0675_4c59_a0f4_ab94b9846d47.slice/crio-2b51b5a2ef008ea6d207de9b4e15eca197d01e43c991960fc75e9f676d13fb0f WatchSource:0}: Error finding container 2b51b5a2ef008ea6d207de9b4e15eca197d01e43c991960fc75e9f676d13fb0f: Status 404 returned error can't find the container with id 2b51b5a2ef008ea6d207de9b4e15eca197d01e43c991960fc75e9f676d13fb0f Nov 02 14:22:00.880244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:00.880214 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" event={"ID":"7a879895-0675-4c59-a0f4-ab94b9846d47","Type":"ContainerStarted","Data":"2b51b5a2ef008ea6d207de9b4e15eca197d01e43c991960fc75e9f676d13fb0f"} Nov 02 14:22:01.074498 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:22:01.074463 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:22:01.888453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:01.888410 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" event={"ID":"7a879895-0675-4c59-a0f4-ab94b9846d47","Type":"ContainerStarted","Data":"77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc"} Nov 02 14:22:01.888943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:01.888483 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:01.932875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:01.932817 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" podStartSLOduration=2.932803512 podStartE2EDuration="2.932803512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:01.930116176 +0000 UTC m=+2270.655737948" watchObservedRunningTime="2026-11-02 14:22:01.932803512 +0000 UTC m=+2270.658425281" Nov 02 14:22:02.083726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:02.083687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:03.780907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:03.780865 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:04.522918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:04.522879 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:05.482882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:05.482843 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:05.970312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:05.970251 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:06.300361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.300275 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq"] Nov 02 14:22:06.305730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.305702 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.323563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.323531 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a02c35-openshift-default-dockercfg-bm9cg\"" Nov 02 14:22:06.341607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.341573 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq"] Nov 02 14:22:06.474028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.473967 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.474434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.474408 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.474625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.474607 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.474746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.474731 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2w59\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-kube-api-access-g2w59\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.474875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.474863 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.474975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.474963 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.475128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.475115 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.475228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.475217 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.475485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.475467 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.491084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.491037 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:06.580953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.580857 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.581254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.581227 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.581403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.581389 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.581559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.581545 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.581803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.581786 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.581973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.581958 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.582236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.582214 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w59\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-kube-api-access-g2w59\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.582399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.582384 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.582518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.582504 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.583573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.583548 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.585430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.585404 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.595673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.585782 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.595885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.588484 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.596032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.589069 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.597476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.597450 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.598438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.598415 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.624461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.624409 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2w59\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-kube-api-access-g2w59\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.627676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.627634 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:06.886023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.885969 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:06.924224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:06.924189 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:07.241073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:07.235619 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:07.272509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:07.272483 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq"] Nov 02 14:22:07.273167 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:22:07.273139 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf95dcd7_f3e2_4ced_ac82_9e0e58513b3c.slice/crio-717b6d155fdfa73d612d3a25746ecc9c6caac7d028120d742681ed7bcecfe903 WatchSource:0}: Error finding container 717b6d155fdfa73d612d3a25746ecc9c6caac7d028120d742681ed7bcecfe903: Status 404 returned error can't find the container with id 717b6d155fdfa73d612d3a25746ecc9c6caac7d028120d742681ed7bcecfe903 Nov 02 14:22:07.276184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:07.276149 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:07.276300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:07.276229 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:07.276300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:07.276277 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:07.930283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:07.930245 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" event={"ID":"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c","Type":"ContainerStarted","Data":"f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d"} Nov 02 14:22:07.930705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:07.930301 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" event={"ID":"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c","Type":"ContainerStarted","Data":"717b6d155fdfa73d612d3a25746ecc9c6caac7d028120d742681ed7bcecfe903"} Nov 02 14:22:08.002690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:08.002638 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podStartSLOduration=2.002624103 podStartE2EDuration="2.002624103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:07.989466058 +0000 UTC m=+2276.715087819" watchObservedRunningTime="2026-11-02 14:22:08.002624103 +0000 UTC m=+2276.728245873" Nov 02 14:22:08.791558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:08.791525 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:08.925078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:08.925045 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:08.931412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:08.931389 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:08.937117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:08.937090 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:08.937888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:08.937863 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:22:08.955240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:08.955213 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:09.175576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:09.175539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:10.525186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:10.525126 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:10.725817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:10.725787 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:10.917263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:10.917225 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:12.302138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.302100 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:12.479876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.479841 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:12.620129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.620032 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9"] Nov 02 14:22:12.687258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.687221 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9"] Nov 02 14:22:12.688102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.688080 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:12.768820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.768600 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a02c35\"" Nov 02 14:22:12.770738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.769257 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a02c35-dockercfg-fx5cn\"" Nov 02 14:22:12.790269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.790153 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9gqh\" (UniqueName: \"kubernetes.io/projected/40d0620c-2792-4b2e-8c86-367582a82e2c-kube-api-access-f9gqh\") pod \"payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9\" (UID: \"40d0620c-2792-4b2e-8c86-367582a82e2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:12.790269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.790212 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d0620c-2792-4b2e-8c86-367582a82e2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9\" (UID: \"40d0620c-2792-4b2e-8c86-367582a82e2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:12.891386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.891349 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9gqh\" (UniqueName: \"kubernetes.io/projected/40d0620c-2792-4b2e-8c86-367582a82e2c-kube-api-access-f9gqh\") pod \"payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9\" (UID: \"40d0620c-2792-4b2e-8c86-367582a82e2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:12.891596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.891393 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d0620c-2792-4b2e-8c86-367582a82e2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9\" (UID: \"40d0620c-2792-4b2e-8c86-367582a82e2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:12.892254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.892224 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d0620c-2792-4b2e-8c86-367582a82e2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9\" (UID: \"40d0620c-2792-4b2e-8c86-367582a82e2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:12.898374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.898351 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:22:12.917385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:12.917358 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9gqh\" (UniqueName: \"kubernetes.io/projected/40d0620c-2792-4b2e-8c86-367582a82e2c-kube-api-access-f9gqh\") pod \"payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9\" (UID: \"40d0620c-2792-4b2e-8c86-367582a82e2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:13.082992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:13.082946 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:13.180863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:13.179422 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:13.460079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:13.459944 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9"] Nov 02 14:22:13.463677 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:22:13.463644 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d0620c_2792_4b2e_8c86_367582a82e2c.slice/crio-b2825b222eec1dbc04593c1d37271694dd2943ad7c81b2a5e00a34d94c575a50 WatchSource:0}: Error finding container b2825b222eec1dbc04593c1d37271694dd2943ad7c81b2a5e00a34d94c575a50: Status 404 returned error can't find the container with id b2825b222eec1dbc04593c1d37271694dd2943ad7c81b2a5e00a34d94c575a50 Nov 02 14:22:13.999705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:13.999668 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" event={"ID":"40d0620c-2792-4b2e-8c86-367582a82e2c","Type":"ContainerStarted","Data":"f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632"} Nov 02 14:22:13.999893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:13.999732 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:13.999893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:13.999756 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" event={"ID":"40d0620c-2792-4b2e-8c86-367582a82e2c","Type":"ContainerStarted","Data":"b2825b222eec1dbc04593c1d37271694dd2943ad7c81b2a5e00a34d94c575a50"} Nov 02 14:22:14.051390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:14.051280 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" podStartSLOduration=2.051258326 podStartE2EDuration="2.051258326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:14.050798975 +0000 UTC m=+2282.776420746" watchObservedRunningTime="2026-11-02 14:22:14.051258326 +0000 UTC m=+2282.776880101" Nov 02 14:22:14.880900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:14.880854 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:16.484926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:16.484881 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:18.267677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:18.267635 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:19.969933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:19.969896 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:21.475686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:21.475578 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:21.640731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:21.640689 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:21.827725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:21.827627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:22.057269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:22.057236 5591 generic.go:356] "Generic (PLEG): container finished" podID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerID="6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca" exitCode=0 Nov 02 14:22:22.057448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:22.057309 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" event={"ID":"8845a851-daf1-4d64-9341-ea6b4d256ce5","Type":"ContainerDied","Data":"6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca"} Nov 02 14:22:22.057738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:22.057724 5591 scope.go:122] "RemoveContainer" containerID="6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca" Nov 02 14:22:22.059553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:22.059526 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:22.059629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:22.059593 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:22.059667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:22.059629 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:23.066480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:23.066447 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" event={"ID":"8845a851-daf1-4d64-9341-ea6b4d256ce5","Type":"ContainerStarted","Data":"9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b"} Nov 02 14:22:23.158250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:23.158211 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:22:23.158482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:23.158417 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 14:22:23.158557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:23.158481 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 14:22:23.273479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:23.273441 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:24.157703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:24.157668 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 14:22:24.158178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:24.157727 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 14:22:24.376798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:24.376765 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:25.009479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:25.009451 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:22:25.087377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:25.087334 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:25.162196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:25.162147 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 14:22:25.162703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:25.162266 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 14:22:26.081008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:26.080721 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:22:26.179008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:26.169857 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 14:22:26.179008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:26.169943 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 14:22:26.931139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:26.931100 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:27.157406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:27.157370 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 14:22:27.157562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:27.157428 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 14:22:28.158208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:28.158175 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 02 14:22:28.158679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:28.158241 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 02 14:22:28.757811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:28.757774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:29.163218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:29.163180 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:22:29.163718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:29.163568 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:22:29.164271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:29.164252 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:22:30.441057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:30.441017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:31.074408 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:22:31.074370 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:22:32.043917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:32.043880 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:32.589696 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:32.589613 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:33.687040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:33.685407 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:33.853455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:33.853414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:34.016014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:34.015901 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:34.420373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:34.420340 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:34.609747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:34.609707 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:35.538555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:35.538523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:35.711534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:35.711423 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:35.878253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:35.878168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:37.407006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:37.395712 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:38.543241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:38.543206 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:38.704232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:38.704199 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:39.282423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:39.282365 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:40.190256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.190221 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:40.355741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.355689 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:40.493054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.470900 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd"] Nov 02 14:22:40.546842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.546804 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd"] Nov 02 14:22:40.547265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.547228 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:40.604130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.604098 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bc34db\"" Nov 02 14:22:40.604691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.604666 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bc34db-dockercfg-p9nt5\"" Nov 02 14:22:40.640363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.640302 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd\" (UID: \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:40.640686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.640664 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hb9\" (UniqueName: \"kubernetes.io/projected/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-kube-api-access-x7hb9\") pod \"payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd\" (UID: \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:40.742324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.742282 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd\" (UID: \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:40.742530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.742358 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hb9\" (UniqueName: \"kubernetes.io/projected/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-kube-api-access-x7hb9\") pod \"payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd\" (UID: \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:40.743090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.743034 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd\" (UID: \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:40.767885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.767855 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hb9\" (UniqueName: \"kubernetes.io/projected/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-kube-api-access-x7hb9\") pod \"payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd\" (UID: \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:40.932765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:40.932733 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:41.091418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:41.091206 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:41.238020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:41.237976 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd"] Nov 02 14:22:41.238492 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:22:41.238464 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc2c0ac_28c8_4708_b02e_eb77d97fbe77.slice/crio-8e61f567b045450fe7fdab7382fc3c9b62402656afd1d3f45f2a08ca0ab3f6d2 WatchSource:0}: Error finding container 8e61f567b045450fe7fdab7382fc3c9b62402656afd1d3f45f2a08ca0ab3f6d2: Status 404 returned error can't find the container with id 8e61f567b045450fe7fdab7382fc3c9b62402656afd1d3f45f2a08ca0ab3f6d2 Nov 02 14:22:41.305579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:41.305546 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" event={"ID":"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77","Type":"ContainerStarted","Data":"8e61f567b045450fe7fdab7382fc3c9b62402656afd1d3f45f2a08ca0ab3f6d2"} Nov 02 14:22:42.315110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:42.315076 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" event={"ID":"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77","Type":"ContainerStarted","Data":"84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4"} Nov 02 14:22:42.315548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:42.315219 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:42.358548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:42.358488 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" podStartSLOduration=2.358475194 podStartE2EDuration="2.358475194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:42.356685392 +0000 UTC m=+2311.082307163" watchObservedRunningTime="2026-11-02 14:22:42.358475194 +0000 UTC m=+2311.084097022" Nov 02 14:22:42.761948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:42.759024 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:44.380893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:44.380854 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:46.062668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:46.062631 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:47.701017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:47.696650 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:49.551879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:49.551717 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:49.873054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:49.872933 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:51.057864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:51.057811 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:51.478081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:51.477546 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:52.308759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:52.308722 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:53.290960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:53.287015 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:53.324496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:53.324469 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:22:55.066515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:55.066472 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:56.641184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:56.640962 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:58.475329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:58.472933 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:22:58.479663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:58.479627 5591 generic.go:356] "Generic (PLEG): container finished" podID="98702400-9887-4120-adea-809091fdf0aa" containerID="27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f" exitCode=0 Nov 02 14:22:58.479832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:58.479701 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" event={"ID":"98702400-9887-4120-adea-809091fdf0aa","Type":"ContainerDied","Data":"27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f"} Nov 02 14:22:58.480228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:58.480205 5591 scope.go:122] "RemoveContainer" containerID="27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f" Nov 02 14:22:58.482251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:58.482219 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:58.482356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:58.482312 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:58.482423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:58.482362 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:22:59.488769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:59.488737 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" event={"ID":"98702400-9887-4120-adea-809091fdf0aa","Type":"ContainerStarted","Data":"5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a"} Nov 02 14:22:59.701056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:59.701026 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:22:59.764779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:59.764693 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:22:59.765892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:59.765865 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 14:22:59.766063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:22:59.765915 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 14:23:00.162973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:00.162935 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:00.765442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:00.765404 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 14:23:00.765879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:00.765487 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 14:23:01.082542 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:23:01.082458 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:23:01.765631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:01.765584 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 14:23:01.766100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:01.765643 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 14:23:01.960843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:01.960771 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:02.765498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:02.765464 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 14:23:02.765692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:02.765524 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 14:23:03.743017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:03.742270 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:03.786035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:03.781111 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 02 14:23:03.786035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:03.781186 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 02 14:23:04.924215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:04.924176 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:23:04.926170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:04.926146 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:23:05.641012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:05.636047 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:07.033403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:07.033364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:07.410657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:07.410620 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:08.306997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:08.306945 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:08.476186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:08.476121 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:08.637607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:08.637578 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:08.921050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:08.920931 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:09.107933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:09.107868 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:09.288809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:09.288680 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:09.807139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:09.807090 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:10.022009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:10.016442 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:11.148635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:11.148581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:11.324793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:11.324754 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:12.858995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:12.858942 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:14.493362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:14.493315 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:15.160021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:15.156559 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:23:16.161587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:16.161550 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:17.858090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:17.858053 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:18.023100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:18.023065 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:18.179358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:18.179327 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:19.650018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:19.649043 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:19.899385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:19.899348 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:20.063291 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:20.063187 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:20.253192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:20.253155 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:21.227571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:21.227536 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:21.872007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:21.869647 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:23.547939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:23.547847 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:24.495321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:24.495259 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:24.978134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:24.978097 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:23:24.978590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:24.978183 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:25.114437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:25.114362 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:26.814459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:26.814426 5591 generic.go:356] "Generic (PLEG): container finished" podID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerID="f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d" exitCode=0 Nov 02 14:23:26.814913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:26.814499 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" event={"ID":"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c","Type":"ContainerDied","Data":"f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d"} Nov 02 14:23:26.814993 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:26.814964 5591 scope.go:122] "RemoveContainer" containerID="f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d" Nov 02 14:23:26.816830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:26.816803 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:23:26.816895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:26.816878 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:23:26.816933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:26.816911 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:23:27.114452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:27.114361 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:27.827755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:27.827717 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" event={"ID":"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c","Type":"ContainerStarted","Data":"c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2"} Nov 02 14:23:27.924543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:27.924503 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:23:27.924729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:27.924665 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 14:23:27.924775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:27.924723 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 14:23:28.638011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:28.635544 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:28.925306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:28.925221 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 14:23:28.925306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:28.925280 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 14:23:29.925367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:29.925326 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 14:23:29.925781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:29.925393 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 14:23:30.391063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:30.387491 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:30.925170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:30.925139 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 14:23:30.925350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:30.925188 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 14:23:31.082123 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:23:31.082085 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:23:31.925520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:31.925485 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 14:23:31.925725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:31.925542 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 14:23:32.073714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:32.073676 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:32.925268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:32.925236 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 14:23:32.925746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:32.925304 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 14:23:33.771193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:33.771119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:33.977009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:33.976149 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:34.176066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:34.176034 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:23:34.177330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:34.177297 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:23:34.178172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:34.178145 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:23:34.577320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:34.577236 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:35.765580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:35.765542 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:37.474851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:37.474812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:37.753058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:37.746934 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:38.517408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:38.517374 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:38.788387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:38.788297 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:38.980092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:38.980054 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:39.174172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:39.174137 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:39.434057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:39.433748 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:39.682118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:39.682073 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:39.878736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:39.878602 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:40.492050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.492021 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:40.622098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.622062 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b"] Nov 02 14:23:40.626484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.626464 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:40.651454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.651424 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-605541\"" Nov 02 14:23:40.668152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.668118 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b"] Nov 02 14:23:40.708482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.708446 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-605541-dockercfg-s249f\"" Nov 02 14:23:40.744058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.743800 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33bbdce-378e-4067-8323-3a1a9892f2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b\" (UID: \"d33bbdce-378e-4067-8323-3a1a9892f2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:40.744058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.743886 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vss49\" (UniqueName: \"kubernetes.io/projected/d33bbdce-378e-4067-8323-3a1a9892f2e6-kube-api-access-vss49\") pod \"payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b\" (UID: \"d33bbdce-378e-4067-8323-3a1a9892f2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:40.805648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.805603 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:40.887009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.847721 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33bbdce-378e-4067-8323-3a1a9892f2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b\" (UID: \"d33bbdce-378e-4067-8323-3a1a9892f2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:40.887009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.854202 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33bbdce-378e-4067-8323-3a1a9892f2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b\" (UID: \"d33bbdce-378e-4067-8323-3a1a9892f2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:40.887009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.883609 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vss49\" (UniqueName: \"kubernetes.io/projected/d33bbdce-378e-4067-8323-3a1a9892f2e6-kube-api-access-vss49\") pod \"payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b\" (UID: \"d33bbdce-378e-4067-8323-3a1a9892f2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:40.925355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.925324 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vss49\" (UniqueName: \"kubernetes.io/projected/d33bbdce-378e-4067-8323-3a1a9892f2e6-kube-api-access-vss49\") pod \"payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b\" (UID: \"d33bbdce-378e-4067-8323-3a1a9892f2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:40.942998 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:40.942958 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:41.256796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.256765 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b"] Nov 02 14:23:41.256953 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:23:41.256855 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33bbdce_378e_4067_8323_3a1a9892f2e6.slice/crio-b27906c8a61e6b33387a403ece50d68fb6e2c5f7c0a835808ea3c5971ef07595 WatchSource:0}: Error finding container b27906c8a61e6b33387a403ece50d68fb6e2c5f7c0a835808ea3c5971ef07595: Status 404 returned error can't find the container with id b27906c8a61e6b33387a403ece50d68fb6e2c5f7c0a835808ea3c5971ef07595 Nov 02 14:23:41.524554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.524469 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:41.722999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.722958 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh"] Nov 02 14:23:41.727739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.727713 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.749283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.749259 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"routing-overlay\"" Nov 02 14:23:41.749454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.749402 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a02c35\"" Nov 02 14:23:41.749454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.749435 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"openshift-service-ca.crt\"" Nov 02 14:23:41.749716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.749704 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"kube-root-ca.crt\"" Nov 02 14:23:41.766555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.766527 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh"] Nov 02 14:23:41.800958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.800879 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.800958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.800938 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.801282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.801252 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.801346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.801310 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st5wk\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-kube-api-access-st5wk\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.827251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.827219 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-e2e--ac2d6d70-dockercfg-gjz62\"" Nov 02 14:23:41.827251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.827237 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"agc-cred-addf94a36aad\"" Nov 02 14:23:41.922009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.917352 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.922009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.917463 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.922009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.917666 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.922009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.917727 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st5wk\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-kube-api-access-st5wk\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.922009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.919853 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.933800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.933755 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.973002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.961738 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st5wk\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-kube-api-access-st5wk\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:41.980001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:41.977554 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:42.046007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:42.042730 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:42.046007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:42.043813 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:42.046007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:42.045344 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" event={"ID":"d33bbdce-378e-4067-8323-3a1a9892f2e6","Type":"ContainerStarted","Data":"d1b482f6cab27d6da149f6cd8cd20272a95877a3d5883331affa8a7e04484c0f"} Nov 02 14:23:42.046007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:42.045403 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" event={"ID":"d33bbdce-378e-4067-8323-3a1a9892f2e6","Type":"ContainerStarted","Data":"b27906c8a61e6b33387a403ece50d68fb6e2c5f7c0a835808ea3c5971ef07595"} Nov 02 14:23:42.046007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:42.045968 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:42.132635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:42.132569 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" podStartSLOduration=2.132550086 podStartE2EDuration="2.132550086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:42.129029367 +0000 UTC m=+2370.854651137" watchObservedRunningTime="2026-11-02 14:23:42.132550086 +0000 UTC m=+2370.858171857" Nov 02 14:23:42.415647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:42.415615 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh"] Nov 02 14:23:42.416416 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:23:42.416391 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc410a78_1afb_44cc_a1a6_c06ffa7a7589.slice/crio-0bb5a66de1c3c9b9ac145573a1105789b8c689118aac95477e67717683e437b8 WatchSource:0}: Error finding container 0bb5a66de1c3c9b9ac145573a1105789b8c689118aac95477e67717683e437b8: Status 404 returned error can't find the container with id 0bb5a66de1c3c9b9ac145573a1105789b8c689118aac95477e67717683e437b8 Nov 02 14:23:43.053184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:43.053141 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" event={"ID":"dc410a78-1afb-44cc-a1a6-c06ffa7a7589","Type":"ContainerStarted","Data":"88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2"} Nov 02 14:23:43.053682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:43.053224 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" event={"ID":"dc410a78-1afb-44cc-a1a6-c06ffa7a7589","Type":"ContainerStarted","Data":"0bb5a66de1c3c9b9ac145573a1105789b8c689118aac95477e67717683e437b8"} Nov 02 14:23:43.053682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:43.053364 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:43.096394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:43.096346 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" podStartSLOduration=2.096334899 podStartE2EDuration="2.096334899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:43.095572786 +0000 UTC m=+2371.821194555" watchObservedRunningTime="2026-11-02 14:23:43.096334899 +0000 UTC m=+2371.821956668" Nov 02 14:23:43.753009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:43.752630 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:45.464667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:45.461693 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:47.095723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:47.095687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:48.873518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:48.873475 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:50.466706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:50.461320 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:52.056040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:52.053122 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:53.056967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:53.056937 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:23:53.779060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:53.776259 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:54.063317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:54.063236 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:23:55.494883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:55.494840 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:55.682861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:55.682819 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:55.848527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:55.848441 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:56.476455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:56.476422 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:57.308685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:57.308636 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:23:59.077138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:23:59.077106 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:00.982437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:00.982400 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:01.075651 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:24:01.075617 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:24:02.648941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:02.648901 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:02.810773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:02.810738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:04.472803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:04.472663 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:04.885465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:04.885433 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:24:05.429236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:05.427337 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:06.174849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:06.170939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:06.562274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:06.562185 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:06.727036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:06.727002 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:06.896916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:06.896882 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:07.118608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:07.118522 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:07.310338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:07.310254 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:07.899936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:07.899905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:08.091008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:08.090378 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:09.410314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:09.410276 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:09.678153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:09.678075 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:10.016798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:10.016711 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:11.389138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:11.389095 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:11.967091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:11.967055 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:24:11.993412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:11.993387 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:24:12.832471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:12.832433 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:13.588037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:13.575523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:15.268624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:15.267270 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:16.861022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:16.860917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:18.523421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:18.523382 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:18.687779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:18.687734 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:18.883562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:18.883529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:19.110973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:19.110923 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:19.286483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:19.286402 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:19.909354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:19.909304 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:20.300420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:20.300334 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:20.710059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:20.710016 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:22.282250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:22.282216 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:22.540858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:22.540786 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:22.772460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:22.772422 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:22.963860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:22.963822 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:24.146647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:24.146606 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:24.615968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:24.615882 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:25.878939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:25.878888 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:26.296587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:26.296555 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:26.501306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:26.501270 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:27.873200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:27.872532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:28.071184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:28.071152 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:28.263565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:28.263513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:30.684687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:30.684654 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:31.074006 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:24:31.073898 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:24:31.083224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:31.083198 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:31.260120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:31.260083 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:31.450918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:31.450844 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:31.735923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:31.735842 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:32.203453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:32.203418 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:32.376601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:32.376561 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:33.027505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:33.027468 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:33.204546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:33.204509 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:33.523442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:33.523382 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:35.268009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:35.267363 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:36.856004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:36.855625 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:38.595437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:38.593505 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:40.501914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:40.501879 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:40.679060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:40.679020 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:42.124161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:42.124089 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:43.775008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:43.772070 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:45.373754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:45.373702 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:47.031492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:47.031416 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:48.648662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:48.648621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:50.278056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:50.274924 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:51.873009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:51.865595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:53.543395 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:53.543211 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:55.143808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:55.143763 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:56.905101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:56.905059 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:24:56.909020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:56.908969 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:24:58.649007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:24:58.643873 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:00.404997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:00.404892 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:00.740064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:00.739954 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:00.941320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:00.941270 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:01.156859 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:25:01.156769 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:25:01.166239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:01.166208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:02.253360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:02.253325 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:02.427524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:02.427492 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:03.975113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:03.975080 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg"] Nov 02 14:25:03.979842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:03.979821 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.011595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.011561 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-821b4dcc9b92\"" Nov 02 14:25:04.011595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.011562 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-b1ab1ce91238\"" Nov 02 14:25:04.011840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.011737 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ee3eca\"" Nov 02 14:25:04.011840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.011794 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-844ee05b4018\"" Nov 02 14:25:04.011840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.011805 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"routing-overlay\"" Nov 02 14:25:04.012042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.011836 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"kube-root-ca.crt\"" Nov 02 14:25:04.012042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.011874 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-e2e--6a9d33b0-dockercfg-b6wf9\"" Nov 02 14:25:04.012042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.011911 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"openshift-service-ca.crt\"" Nov 02 14:25:04.017637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.017607 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg"] Nov 02 14:25:04.147753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.147718 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.147957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.147766 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.148058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.147956 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qrs\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-kube-api-access-b7qrs\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.148230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.148212 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.249806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.249705 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.249806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.249801 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.250087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.249844 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.250087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.249928 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qrs\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-kube-api-access-b7qrs\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.250518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.250486 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.250682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.250588 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.252520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.252501 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.275396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.275371 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qrs\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-kube-api-access-b7qrs\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.294715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.294688 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.373113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.373003 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:04.656682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.656653 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg"] Nov 02 14:25:04.657525 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:25:04.657499 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1e0ac2_5d9a_410b_8b93_fefb45bd409d.slice/crio-790299a251c4e43343dc9e6c61e490eb93bcc792a2c956d38555391311f4ec1e WatchSource:0}: Error finding container 790299a251c4e43343dc9e6c61e490eb93bcc792a2c956d38555391311f4ec1e: Status 404 returned error can't find the container with id 790299a251c4e43343dc9e6c61e490eb93bcc792a2c956d38555391311f4ec1e Nov 02 14:25:04.789669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.789629 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" event={"ID":"af1e0ac2-5d9a-410b-8b93-fefb45bd409d","Type":"ContainerStarted","Data":"a580253a3f2037b7db0261713b5a20f3ffa760729e57f918814ab576c0119eca"} Nov 02 14:25:04.789872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.789696 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" event={"ID":"af1e0ac2-5d9a-410b-8b93-fefb45bd409d","Type":"ContainerStarted","Data":"790299a251c4e43343dc9e6c61e490eb93bcc792a2c956d38555391311f4ec1e"} Nov 02 14:25:04.789872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.789720 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:04.832726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:04.832676 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" podStartSLOduration=1.832663428 podStartE2EDuration="1.832663428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:04.831500542 +0000 UTC m=+2453.557122311" watchObservedRunningTime="2026-11-02 14:25:04.832663428 +0000 UTC m=+2453.558285197" Nov 02 14:25:06.078883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:06.070446 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:07.712158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:07.712123 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:07.904023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:07.903948 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:09.480051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:09.469716 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:11.139945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:11.139905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:12.991122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:12.991082 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:14.742595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:14.742553 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:15.311649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:15.311614 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh"] Nov 02 14:25:15.380486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:15.380431 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:15.800677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:15.800640 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:25:16.470008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:16.466724 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:16.631129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:16.631092 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:18.159969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:18.159930 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:19.799026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:19.798971 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:20.312256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.312213 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" podUID="dc410a78-1afb-44cc-a1a6-c06ffa7a7589" containerName="payload-processing" containerID="cri-o://88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2" gracePeriod=25 Nov 02 14:25:20.597048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.597016 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:25:20.652558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.652518 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-kube-api-access-st5wk\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-kube-api-access-st5wk\") pod \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " Nov 02 14:25:20.653145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.653106 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-routing-overlay-volume\") pod \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " Nov 02 14:25:20.653300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.653195 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-plugins-config-volume\") pod \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " Nov 02 14:25:20.653395 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.653331 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-provider-credentials\") pod \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\" (UID: \"dc410a78-1afb-44cc-a1a6-c06ffa7a7589\") " Nov 02 14:25:20.653483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.653451 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-routing-overlay-volume" pod "dc410a78-1afb-44cc-a1a6-c06ffa7a7589" (UID: "dc410a78-1afb-44cc-a1a6-c06ffa7a7589"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:20.653771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.653744 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-plugins-config-volume" pod "dc410a78-1afb-44cc-a1a6-c06ffa7a7589" (UID: "dc410a78-1afb-44cc-a1a6-c06ffa7a7589"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:20.654019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.653954 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:25:20.654019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.653996 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:25:20.656135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.656107 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-kube-api-access-st5wk" pod "dc410a78-1afb-44cc-a1a6-c06ffa7a7589" (UID: "dc410a78-1afb-44cc-a1a6-c06ffa7a7589"). InnerVolumeSpecName "kube-api-access-st5wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:20.656242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.656180 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-provider-credentials" pod "dc410a78-1afb-44cc-a1a6-c06ffa7a7589" (UID: "dc410a78-1afb-44cc-a1a6-c06ffa7a7589"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:20.755524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.755488 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:25:20.755524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:20.755519 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st5wk\" (UniqueName: \"kubernetes.io/projected/dc410a78-1afb-44cc-a1a6-c06ffa7a7589-kube-api-access-st5wk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:25:21.038942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.038909 5591 generic.go:356] "Generic (PLEG): container finished" podID="dc410a78-1afb-44cc-a1a6-c06ffa7a7589" containerID="88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2" exitCode=0 Nov 02 14:25:21.039432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.039151 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" event={"ID":"dc410a78-1afb-44cc-a1a6-c06ffa7a7589","Type":"ContainerDied","Data":"88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2"} Nov 02 14:25:21.039573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.039558 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" event={"ID":"dc410a78-1afb-44cc-a1a6-c06ffa7a7589","Type":"ContainerDied","Data":"0bb5a66de1c3c9b9ac145573a1105789b8c689118aac95477e67717683e437b8"} Nov 02 14:25:21.039657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.039645 5591 scope.go:122] "RemoveContainer" containerID="88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2" Nov 02 14:25:21.039788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.039240 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh" Nov 02 14:25:21.054331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.054306 5591 scope.go:122] "RemoveContainer" containerID="88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2" Nov 02 14:25:21.054629 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:25:21.054608 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2\": container with ID starting with 88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2 not found: ID does not exist" containerID="88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2" Nov 02 14:25:21.054703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.054636 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2"} err="failed to get container status \"88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2\": rpc error: code = NotFound desc = could not find container \"88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2\": container with ID starting with 88603496749c48b1defb9b01b502461d6c7a134f4924ec65ab161ba46cd5bad2 not found: ID does not exist" Nov 02 14:25:21.094952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.094919 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh"] Nov 02 14:25:21.108904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.108867 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9bxbxh"] Nov 02 14:25:21.433689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.433647 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc410a78-1afb-44cc-a1a6-c06ffa7a7589" path="/var/lib/kubelet/pods/dc410a78-1afb-44cc-a1a6-c06ffa7a7589/volumes" Nov 02 14:25:21.496847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:21.490331 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:23.347928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:23.347890 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:24.582361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:24.582331 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:24.746773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:24.746739 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:25.212052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:25.211927 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:27.004112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:27.000405 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:28.586006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:28.582788 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:28.959414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:28.959362 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:29.274056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:29.273049 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:29.442724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:29.442692 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:29.998723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:29.998681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:30.159994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:30.159943 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:30.431020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:30.426464 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:31.075333 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:25:31.075287 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:25:32.090775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:32.090708 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:33.748050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:33.748008 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:35.350598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:35.350552 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:36.925540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:36.925502 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:38.660619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:38.660567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:40.466119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:40.466077 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:42.099382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:42.099341 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:42.334457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:42.334409 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:42.502421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:42.502385 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:42.971319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:42.971244 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:43.682519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:43.682481 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:45.369180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:45.368542 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:46.960629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:46.960156 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:48.672522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:48.672486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:49.673424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:49.673385 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:49.842495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:49.842454 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:50.369209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:50.369168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:52.054014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:52.046470 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:52.934833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:52.934776 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:53.962590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:53.962542 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:54.406715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:54.406675 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:54.901554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:54.901520 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:55.068007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:55.065461 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:55.817498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:55.817466 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:55.852627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:55.852592 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:25:57.345122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:57.345088 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:57.558004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:57.551172 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:57.747833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:57.747780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:58.004593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:58.004517 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:58.185928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:58.185891 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:58.369068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:58.368966 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:58.560386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:58.560349 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:59.367010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:59.366199 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:25:59.634067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:25:59.629420 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:00.369567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:00.369533 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:00.919908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:00.919872 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:01.079650 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:01.079609 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:26:01.584568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:01.584533 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:01.757298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:01.757264 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:02.535097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:02.535044 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:02.874362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:02.874256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:03.080247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.080214 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf"] Nov 02 14:26:03.081063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.081015 5591 cpu_manager.go:414] "containerMap: removing container" podUID="dc410a78-1afb-44cc-a1a6-c06ffa7a7589" containerName="payload-processing" Nov 02 14:26:03.081063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.081038 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc410a78-1afb-44cc-a1a6-c06ffa7a7589" containerName="payload-processing" Nov 02 14:26:03.081297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.081239 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc410a78-1afb-44cc-a1a6-c06ffa7a7589" containerName="payload-processing" Nov 02 14:26:03.085466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.085431 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.104127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.104099 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf"] Nov 02 14:26:03.113041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.112874 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"agc-cred-714f726da461\"" Nov 02 14:26:03.113041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.112959 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-87efd3\"" Nov 02 14:26:03.114048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.113827 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"openshift-service-ca.crt\"" Nov 02 14:26:03.114626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.114601 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"kube-root-ca.crt\"" Nov 02 14:26:03.114754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.114665 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"routing-overlay\"" Nov 02 14:26:03.114754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.114665 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"payload-processing-external-model-e2e--005c2a0a-dockercfg-kf676\"" Nov 02 14:26:03.252018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.245857 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqrx\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-kube-api-access-cvqrx\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.269005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.245973 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.269005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.259229 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.269005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.259266 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.360913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.360882 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.361195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.361171 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.361571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.361545 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqrx\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-kube-api-access-cvqrx\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.361806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.361784 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.362305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.362256 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.362655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.362625 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.365286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.365264 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.386831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.386800 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqrx\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-kube-api-access-cvqrx\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.402097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.401799 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:03.489575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.489542 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:03.725190 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:26:03.725156 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03421ec9_ff49_464c_8ba8_4e2856bda3ca.slice/crio-807e007847444b103f445dd7660162b574ebb0ab525ffaf6305833aa4881ec0f WatchSource:0}: Error finding container 807e007847444b103f445dd7660162b574ebb0ab525ffaf6305833aa4881ec0f: Status 404 returned error can't find the container with id 807e007847444b103f445dd7660162b574ebb0ab525ffaf6305833aa4881ec0f Nov 02 14:26:03.725512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:03.725234 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf"] Nov 02 14:26:04.207949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:04.207918 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:04.371475 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:04.371438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:04.516518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:04.516413 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" event={"ID":"03421ec9-ff49-464c-8ba8-4e2856bda3ca","Type":"ContainerStarted","Data":"13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff"} Nov 02 14:26:04.516909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:04.516870 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:04.517209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:04.517187 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" event={"ID":"03421ec9-ff49-464c-8ba8-4e2856bda3ca","Type":"ContainerStarted","Data":"807e007847444b103f445dd7660162b574ebb0ab525ffaf6305833aa4881ec0f"} Nov 02 14:26:04.571431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:04.571351 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" podStartSLOduration=1.571332199 podStartE2EDuration="1.571332199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:04.568799408 +0000 UTC m=+2513.294421175" watchObservedRunningTime="2026-11-02 14:26:04.571332199 +0000 UTC m=+2513.296953971" Nov 02 14:26:04.572762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:04.572736 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:04.872800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:04.872694 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:05.044143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.043750 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm"] Nov 02 14:26:05.092079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.092029 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:05.185529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.185493 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm"] Nov 02 14:26:05.185827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.185671 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:05.227727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.211540 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-66bc6d-dockercfg-xkbbb\"" Nov 02 14:26:05.227727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.223351 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-66bc6d\"" Nov 02 14:26:05.248093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.238903 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/bdde9b7a-bec6-4c34-a647-2448a0301ff6-kube-api-access-p92v6\") pod \"payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm\" (UID: \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:05.248093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.239190 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9b7a-bec6-4c34-a647-2448a0301ff6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm\" (UID: \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:05.341169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.341126 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9b7a-bec6-4c34-a647-2448a0301ff6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm\" (UID: \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:05.341367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.341272 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/bdde9b7a-bec6-4c34-a647-2448a0301ff6-kube-api-access-p92v6\") pod \"payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm\" (UID: \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:05.341941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.341920 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9b7a-bec6-4c34-a647-2448a0301ff6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm\" (UID: \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:05.365706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.365666 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/bdde9b7a-bec6-4c34-a647-2448a0301ff6-kube-api-access-p92v6\") pod \"payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm\" (UID: \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:05.597313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.597214 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:05.991499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:05.991459 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm"] Nov 02 14:26:06.342591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:06.332084 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:06.537999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:06.537941 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" event={"ID":"bdde9b7a-bec6-4c34-a647-2448a0301ff6","Type":"ContainerStarted","Data":"49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34"} Nov 02 14:26:06.538199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:06.538022 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" event={"ID":"bdde9b7a-bec6-4c34-a647-2448a0301ff6","Type":"ContainerStarted","Data":"f1d7ca464cd804aaadb668f9520a0548a80d82585309fcd65852e6d67600fc72"} Nov 02 14:26:06.538199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:06.538055 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:06.605616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:06.605507 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" podStartSLOduration=1.605493521 podStartE2EDuration="1.605493521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:06.602139907 +0000 UTC m=+2515.327761677" watchObservedRunningTime="2026-11-02 14:26:06.605493521 +0000 UTC m=+2515.331115290" Nov 02 14:26:07.397032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:07.396996 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c"] Nov 02 14:26:07.397513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:07.397249 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" containerID="cri-o://915ee59361dd4e6c600622f41779911aaeda2fc9514c3172e9005e175fd25119" gracePeriod=30 Nov 02 14:26:08.018046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.017998 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn"] Nov 02 14:26:08.022865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.022837 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.038558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.038531 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm"] Nov 02 14:26:08.061208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.060327 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ce0f55-openshift-default-dockercfg-ghzd9\"" Nov 02 14:26:08.069301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.069272 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.084047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.080430 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:08.101951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.101874 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn"] Nov 02 14:26:08.106376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106343 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4529748-2a47-4b75-b885-0a83f37a0705-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.106531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106427 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cfbc2-ec41-47df-9135-7a8454c039f7-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.106531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106508 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.106650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106568 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.106650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106639 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.106772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106689 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.106848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106768 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4529748-2a47-4b75-b885-0a83f37a0705-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.106848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106795 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.106959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106850 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkm4\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-kube-api-access-zgkm4\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.106959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.106877 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.107123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.107058 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.107123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.107094 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.107236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.107164 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.107236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.107198 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2r66\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-kube-api-access-h2r66\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.107404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.107246 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.107404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.107281 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.107404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.107322 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.107550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.107404 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.108474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.108455 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm"] Nov 02 14:26:08.209158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209114 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkm4\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-kube-api-access-zgkm4\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209158 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.209541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209278 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209311 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209373 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.209541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209404 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2r66\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-kube-api-access-h2r66\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.209541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209443 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209478 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209514 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209586 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209662 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4529748-2a47-4b75-b885-0a83f37a0705-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209727 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cfbc2-ec41-47df-9135-7a8454c039f7-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209750 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209771 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209761 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209845 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209856 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209872 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.209926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209898 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.210576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209958 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.210576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.209963 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4529748-2a47-4b75-b885-0a83f37a0705-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.210576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.210066 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.210576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.210079 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.210576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.210492 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.210878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.210638 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4529748-2a47-4b75-b885-0a83f37a0705-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.210878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.210664 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.211280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.211252 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cfbc2-ec41-47df-9135-7a8454c039f7-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.211611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.211577 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.211948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.211926 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.212190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.212171 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.212508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.212481 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.212671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.212656 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4529748-2a47-4b75-b885-0a83f37a0705-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.233488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.233455 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkm4\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-kube-api-access-zgkm4\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.234977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.234955 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.235094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.235040 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.235094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.235057 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2r66\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-kube-api-access-h2r66\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.374518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.374429 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:08.420384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.420348 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:08.710506 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:26:08.710472 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4529748_2a47_4b75_b885_0a83f37a0705.slice/crio-31131d67094d7d7007e262bd617536fa8523b91ab7531e607000cfb5b9bcf210 WatchSource:0}: Error finding container 31131d67094d7d7007e262bd617536fa8523b91ab7531e607000cfb5b9bcf210: Status 404 returned error can't find the container with id 31131d67094d7d7007e262bd617536fa8523b91ab7531e607000cfb5b9bcf210 Nov 02 14:26:08.710715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.710692 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn"] Nov 02 14:26:08.713594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.713561 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:08.713689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.713629 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:08.713689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.713685 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:08.755164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.755140 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm"] Nov 02 14:26:08.756395 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:26:08.756362 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781cfbc2_ec41_47df_9135_7a8454c039f7.slice/crio-1acd7f744c97d117c63568fcf4d59567dc8dc8a19a010ee7e906e1f45c7ad4a5 WatchSource:0}: Error finding container 1acd7f744c97d117c63568fcf4d59567dc8dc8a19a010ee7e906e1f45c7ad4a5: Status 404 returned error can't find the container with id 1acd7f744c97d117c63568fcf4d59567dc8dc8a19a010ee7e906e1f45c7ad4a5 Nov 02 14:26:08.759648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.759613 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:08.759730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.759701 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:08.759772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:08.759745 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:09.484351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:09.484303 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:09.563623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:09.563577 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" event={"ID":"781cfbc2-ec41-47df-9135-7a8454c039f7","Type":"ContainerStarted","Data":"86b09d85e9e1ceb232ac994a2cc2f3b9947a5a956dc6ab1dcd27d8e1e8b80350"} Nov 02 14:26:09.563820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:09.563632 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" event={"ID":"781cfbc2-ec41-47df-9135-7a8454c039f7","Type":"ContainerStarted","Data":"1acd7f744c97d117c63568fcf4d59567dc8dc8a19a010ee7e906e1f45c7ad4a5"} Nov 02 14:26:09.565482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:09.565445 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" event={"ID":"b4529748-2a47-4b75-b885-0a83f37a0705","Type":"ContainerStarted","Data":"a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1"} Nov 02 14:26:09.565616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:09.565500 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" event={"ID":"b4529748-2a47-4b75-b885-0a83f37a0705","Type":"ContainerStarted","Data":"31131d67094d7d7007e262bd617536fa8523b91ab7531e607000cfb5b9bcf210"} Nov 02 14:26:09.627313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:09.627252 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" podStartSLOduration=1.6272385150000002 podStartE2EDuration="1.627238515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:09.624420576 +0000 UTC m=+2518.350042347" watchObservedRunningTime="2026-11-02 14:26:09.627238515 +0000 UTC m=+2518.352860322" Nov 02 14:26:09.674399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:09.674341 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podStartSLOduration=2.674327488 podStartE2EDuration="2.674327488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:09.67135574 +0000 UTC m=+2518.396977511" watchObservedRunningTime="2026-11-02 14:26:09.674327488 +0000 UTC m=+2518.399949258" Nov 02 14:26:09.885714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:09.885676 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:10.175062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.168531 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf"] Nov 02 14:26:10.190282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.190218 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:10.291521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.290836 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:10.377201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.377168 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:10.378576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.377640 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 14:26:10.378858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.378826 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 14:26:10.421121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.421091 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:10.422659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.422629 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 14:26:10.422800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:10.422689 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 14:26:11.382391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:11.382353 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 14:26:11.383003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:11.382952 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 14:26:11.420764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:11.420650 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 14:26:11.420764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:11.420703 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 14:26:11.780064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:11.770262 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:12.386136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.385150 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 14:26:12.386136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.385212 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 14:26:12.421057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.421026 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 14:26:12.421330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.421303 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 14:26:12.622423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.622389 5591 generic.go:356] "Generic (PLEG): container finished" podID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerID="915ee59361dd4e6c600622f41779911aaeda2fc9514c3172e9005e175fd25119" exitCode=0 Nov 02 14:26:12.622599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.622465 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" event={"ID":"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7","Type":"ContainerDied","Data":"915ee59361dd4e6c600622f41779911aaeda2fc9514c3172e9005e175fd25119"} Nov 02 14:26:12.622599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.622564 5591 scope.go:122] "RemoveContainer" containerID="de630999105654b44dd9a3973428dc8f23ca70d8fbd7ab7336ffcc8129536ee0" Nov 02 14:26:12.882820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.882427 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:26:12.995300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995196 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-socket\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.995300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995250 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-certs\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.995644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995453 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-kube-api-access-6ndnf\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-kube-api-access-6ndnf\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.995644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995521 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-envoy\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.995644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995585 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-token\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-token\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.995644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995624 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-podinfo\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.995945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995698 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-data\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.995945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995776 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-credential-socket\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.995945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.995827 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istiod-ca-cert\") pod \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\" (UID: \"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7\") " Nov 02 14:26:12.996706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.996677 5591 scope.go:122] "RemoveContainer" containerID="915ee59361dd4e6c600622f41779911aaeda2fc9514c3172e9005e175fd25119" Nov 02 14:26:12.997196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.997172 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istiod-ca-cert" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:12.997528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.997501 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-data" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:12.997700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.997650 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-certs" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:12.997849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.997730 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-credential-socket" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:12.997957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:12.997903 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-socket" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:13.000689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.000667 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-envoy" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:13.000689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.000670 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-kube-api-access-6ndnf" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "kube-api-access-6ndnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:13.002359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.002298 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-token" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:13.003060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.002683 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-podinfo" pod "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" (UID: "a3a1f163-c7e8-480c-b88a-06eaa37bc2b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:26:13.099045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.099005 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.099246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.099047 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.099555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.099525 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.099683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.099558 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.099683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.099574 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.099683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.099588 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.099683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.099602 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.099683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.099615 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndnf\" (UniqueName: \"kubernetes.io/projected/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-kube-api-access-6ndnf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.100092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.100071 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:13.378067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.377435 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 14:26:13.378067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.377502 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 14:26:13.421234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.421182 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 14:26:13.421627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.421250 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 14:26:13.517007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.514082 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:13.630737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.630648 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" event={"ID":"a3a1f163-c7e8-480c-b88a-06eaa37bc2b7","Type":"ContainerDied","Data":"7ac5ccbb8d6bc875697905af1daa8b3c67fa473ac4b8ef3a7dbb4441bea0329e"} Nov 02 14:26:13.630737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.630722 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c" Nov 02 14:26:13.719328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.719223 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c"] Nov 02 14:26:13.723015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:13.722978 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-9277c"] Nov 02 14:26:14.378008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:14.376438 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 14:26:14.378008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:14.376509 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 14:26:14.421604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:14.421563 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 14:26:14.422138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:14.421619 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 14:26:14.873008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:14.870702 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:15.052579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.052544 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:15.173008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.169051 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" podUID="03421ec9-ff49-464c-8ba8-4e2856bda3ca" containerName="payload-processing" containerID="cri-o://13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff" gracePeriod=25 Nov 02 14:26:15.378006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.377670 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 14:26:15.378006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.377734 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 14:26:15.423637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.423537 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 14:26:15.424189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.423611 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 14:26:15.491483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.491447 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" path="/var/lib/kubelet/pods/a3a1f163-c7e8-480c-b88a-06eaa37bc2b7/volumes" Nov 02 14:26:15.566781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.566739 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:15.570250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.570226 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:15.702148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.702054 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-routing-overlay-volume\") pod \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " Nov 02 14:26:15.702148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.702123 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-provider-credentials\") pod \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " Nov 02 14:26:15.702414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.702392 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-plugins-config-volume\") pod \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " Nov 02 14:26:15.702525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.702509 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-kube-api-access-cvqrx\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-kube-api-access-cvqrx\") pod \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\" (UID: \"03421ec9-ff49-464c-8ba8-4e2856bda3ca\") " Nov 02 14:26:15.707052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.707021 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-plugins-config-volume" pod "03421ec9-ff49-464c-8ba8-4e2856bda3ca" (UID: "03421ec9-ff49-464c-8ba8-4e2856bda3ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:15.711496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.711444 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-routing-overlay-volume" pod "03421ec9-ff49-464c-8ba8-4e2856bda3ca" (UID: "03421ec9-ff49-464c-8ba8-4e2856bda3ca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:15.739102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.728018 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-kube-api-access-cvqrx" pod "03421ec9-ff49-464c-8ba8-4e2856bda3ca" (UID: "03421ec9-ff49-464c-8ba8-4e2856bda3ca"). InnerVolumeSpecName "kube-api-access-cvqrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:15.739102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.730276 5591 generic.go:356] "Generic (PLEG): container finished" podID="03421ec9-ff49-464c-8ba8-4e2856bda3ca" containerID="13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff" exitCode=0 Nov 02 14:26:15.739102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.730420 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" event={"ID":"03421ec9-ff49-464c-8ba8-4e2856bda3ca","Type":"ContainerDied","Data":"13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff"} Nov 02 14:26:15.739102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.730461 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" event={"ID":"03421ec9-ff49-464c-8ba8-4e2856bda3ca","Type":"ContainerDied","Data":"807e007847444b103f445dd7660162b574ebb0ab525ffaf6305833aa4881ec0f"} Nov 02 14:26:15.739102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.730486 5591 scope.go:122] "RemoveContainer" containerID="13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff" Nov 02 14:26:15.739102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.730582 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf" Nov 02 14:26:15.739102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.732904 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-provider-credentials" pod "03421ec9-ff49-464c-8ba8-4e2856bda3ca" (UID: "03421ec9-ff49-464c-8ba8-4e2856bda3ca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:15.777640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.777585 5591 scope.go:122] "RemoveContainer" containerID="13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff" Nov 02 14:26:15.778080 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:15.778039 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff\": container with ID starting with 13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff not found: ID does not exist" containerID="13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff" Nov 02 14:26:15.778199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.778093 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff"} err="failed to get container status \"13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff\": rpc error: code = NotFound desc = could not find container \"13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff\": container with ID starting with 13da56fd8537bfa7a790c350b6c07654c87e36f29d68b5fa9aa1b337b2376bff not found: ID does not exist" Nov 02 14:26:15.804708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.804670 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqrx\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-kube-api-access-cvqrx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:15.804708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.804710 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:15.804952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.804725 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03421ec9-ff49-464c-8ba8-4e2856bda3ca-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:15.804952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:15.804739 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03421ec9-ff49-464c-8ba8-4e2856bda3ca-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:16.040151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.040122 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:16.079041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.079009 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf"] Nov 02 14:26:16.083090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.083065 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-6f74p62zf"] Nov 02 14:26:16.380808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.380724 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:16.381136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.381112 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:16.381885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.381865 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:26:16.426394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.426362 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:16.427367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.426717 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:16.427490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:16.427471 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:26:17.298034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:17.297940 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:17.434049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:17.434016 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03421ec9-ff49-464c-8ba8-4e2856bda3ca" path="/var/lib/kubelet/pods/03421ec9-ff49-464c-8ba8-4e2856bda3ca/volumes" Nov 02 14:26:17.526614 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:17.526578 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ee3eca/agc-cred-844ee05b4018: secret "agc-cred-844ee05b4018" not found Nov 02 14:26:17.526855 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:17.526834 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg: secret "agc-cred-844ee05b4018" not found Nov 02 14:26:17.527061 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:17.527043 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials podName:af1e0ac2-5d9a-410b-8b93-fefb45bd409d nodeName:}" failed. No retries permitted until 2026-11-02 14:26:18.027020756 +0000 UTC m=+2526.752642526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" (UID: "af1e0ac2-5d9a-410b-8b93-fefb45bd409d") : secret "agc-cred-844ee05b4018" not found Nov 02 14:26:17.549475 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:17.549413 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:26:18.034929 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:18.034895 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ee3eca/agc-cred-844ee05b4018: secret "agc-cred-844ee05b4018" not found Nov 02 14:26:18.034929 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:18.034923 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg: secret "agc-cred-844ee05b4018" not found Nov 02 14:26:18.035170 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:18.035014 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials podName:af1e0ac2-5d9a-410b-8b93-fefb45bd409d nodeName:}" failed. No retries permitted until 2026-11-02 14:26:19.034966481 +0000 UTC m=+2527.760588244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" (UID: "af1e0ac2-5d9a-410b-8b93-fefb45bd409d") : secret "agc-cred-844ee05b4018" not found Nov 02 14:26:18.952008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:18.951539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:19.057479 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:19.057446 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ee3eca/agc-cred-844ee05b4018: secret "agc-cred-844ee05b4018" not found Nov 02 14:26:19.057479 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:19.057474 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg: secret "agc-cred-844ee05b4018" not found Nov 02 14:26:19.057693 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:19.057536 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials podName:af1e0ac2-5d9a-410b-8b93-fefb45bd409d nodeName:}" failed. No retries permitted until 2026-11-02 14:26:21.057522066 +0000 UTC m=+2529.783143827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" (UID: "af1e0ac2-5d9a-410b-8b93-fefb45bd409d") : secret "agc-cred-844ee05b4018" not found Nov 02 14:26:19.537202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:19.537166 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg"] Nov 02 14:26:20.622014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:20.619906 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:21.086963 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:21.086928 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ee3eca/agc-cred-844ee05b4018: secret "agc-cred-844ee05b4018" not found Nov 02 14:26:21.086963 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:21.086961 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg: secret "agc-cred-844ee05b4018" not found Nov 02 14:26:21.087198 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:21.087038 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials podName:af1e0ac2-5d9a-410b-8b93-fefb45bd409d nodeName:}" failed. No retries permitted until 2026-11-02 14:26:25.087023191 +0000 UTC m=+2533.812644952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" (UID: "af1e0ac2-5d9a-410b-8b93-fefb45bd409d") : secret "agc-cred-844ee05b4018" not found Nov 02 14:26:22.471464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:22.470654 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:22.636665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:22.636616 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:24.200732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.200694 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:24.538090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.538003 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" podUID="af1e0ac2-5d9a-410b-8b93-fefb45bd409d" containerName="payload-processing" containerID="cri-o://a580253a3f2037b7db0261713b5a20f3ffa760729e57f918814ab576c0119eca" gracePeriod=25 Nov 02 14:26:24.796145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.796073 5591 generic.go:356] "Generic (PLEG): container finished" podID="af1e0ac2-5d9a-410b-8b93-fefb45bd409d" containerID="a580253a3f2037b7db0261713b5a20f3ffa760729e57f918814ab576c0119eca" exitCode=0 Nov 02 14:26:24.796293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.796147 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" event={"ID":"af1e0ac2-5d9a-410b-8b93-fefb45bd409d","Type":"ContainerDied","Data":"a580253a3f2037b7db0261713b5a20f3ffa760729e57f918814ab576c0119eca"} Nov 02 14:26:24.796293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.796228 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" event={"ID":"af1e0ac2-5d9a-410b-8b93-fefb45bd409d","Type":"ContainerDied","Data":"790299a251c4e43343dc9e6c61e490eb93bcc792a2c956d38555391311f4ec1e"} Nov 02 14:26:24.796293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.796245 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="790299a251c4e43343dc9e6c61e490eb93bcc792a2c956d38555391311f4ec1e" Nov 02 14:26:24.809646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.809619 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw"] Nov 02 14:26:24.810416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.810394 5591 cpu_manager.go:414] "containerMap: removing container" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" Nov 02 14:26:24.810416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.810416 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" Nov 02 14:26:24.810586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.810452 5591 cpu_manager.go:414] "containerMap: removing container" podUID="03421ec9-ff49-464c-8ba8-4e2856bda3ca" containerName="payload-processing" Nov 02 14:26:24.810586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.810460 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03421ec9-ff49-464c-8ba8-4e2856bda3ca" containerName="payload-processing" Nov 02 14:26:24.810586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.810480 5591 cpu_manager.go:414] "containerMap: removing container" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" Nov 02 14:26:24.810586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.810488 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" Nov 02 14:26:24.810798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.810669 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="03421ec9-ff49-464c-8ba8-4e2856bda3ca" containerName="payload-processing" Nov 02 14:26:24.810798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.810683 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" Nov 02 14:26:24.814863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.814843 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.822341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.822320 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:26:24.831684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.831657 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2d13f4-openshift-default-dockercfg-24bpg\"" Nov 02 14:26:24.840545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.840520 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-plugins-config-volume\") pod \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " Nov 02 14:26:24.840678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.840576 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-kube-api-access-b7qrs\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-kube-api-access-b7qrs\") pod \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " Nov 02 14:26:24.840749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.840736 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials\") pod \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " Nov 02 14:26:24.840821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.840808 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-routing-overlay-volume\") pod \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\" (UID: \"af1e0ac2-5d9a-410b-8b93-fefb45bd409d\") " Nov 02 14:26:24.840943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.840918 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-plugins-config-volume" pod "af1e0ac2-5d9a-410b-8b93-fefb45bd409d" (UID: "af1e0ac2-5d9a-410b-8b93-fefb45bd409d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:24.841196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841174 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-routing-overlay-volume" pod "af1e0ac2-5d9a-410b-8b93-fefb45bd409d" (UID: "af1e0ac2-5d9a-410b-8b93-fefb45bd409d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:24.841277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841196 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae3377c-0760-40ea-9259-6edc9e44d021-istiod-ca-cert\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.841277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841245 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-credential-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.841399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841382 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae3377c-0760-40ea-9259-6edc9e44d021-istio-podinfo\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.841470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841432 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-certs\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.841616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841600 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-envoy\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.841688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841650 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-data\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.841751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841725 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-istio-token\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.841819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841791 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.841884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841828 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczxw\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-kube-api-access-tczxw\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.842017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.841967 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:24.842096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.842021 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:24.843401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.843361 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-kube-api-access-b7qrs" pod "af1e0ac2-5d9a-410b-8b93-fefb45bd409d" (UID: "af1e0ac2-5d9a-410b-8b93-fefb45bd409d"). InnerVolumeSpecName "kube-api-access-b7qrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:24.843401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.843388 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials" pod "af1e0ac2-5d9a-410b-8b93-fefb45bd409d" (UID: "af1e0ac2-5d9a-410b-8b93-fefb45bd409d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:24.850468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.850438 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw"] Nov 02 14:26:24.879468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.879435 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:24.943971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.943922 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczxw\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-kube-api-access-tczxw\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944108 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae3377c-0760-40ea-9259-6edc9e44d021-istiod-ca-cert\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944153 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-credential-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944199 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae3377c-0760-40ea-9259-6edc9e44d021-istio-podinfo\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944229 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-certs\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944330 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-envoy\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944385 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-data\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944472 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-istio-token\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944563 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.944708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944643 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:24.944708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.944661 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7qrs\" (UniqueName: \"kubernetes.io/projected/af1e0ac2-5d9a-410b-8b93-fefb45bd409d-kube-api-access-b7qrs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:26:24.945080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.945051 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.945364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.945340 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-data\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.945623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.945595 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae3377c-0760-40ea-9259-6edc9e44d021-istiod-ca-cert\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.945874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.945851 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-credential-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.945978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.945895 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-certs\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.947655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.947629 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae3377c-0760-40ea-9259-6edc9e44d021-istio-podinfo\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.948591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.948567 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-envoy\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.968177 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.968153 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczxw\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-kube-api-access-tczxw\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:24.968756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:24.968737 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-istio-token\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:25.133050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.133022 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:25.466613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.466581 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw"] Nov 02 14:26:25.467219 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:26:25.467195 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae3377c_0760_40ea_9259_6edc9e44d021.slice/crio-33b242a8bfa0dc261b2e0e1e7197e0468fba73e9464919a7754a8c41b6f73199 WatchSource:0}: Error finding container 33b242a8bfa0dc261b2e0e1e7197e0468fba73e9464919a7754a8c41b6f73199: Status 404 returned error can't find the container with id 33b242a8bfa0dc261b2e0e1e7197e0468fba73e9464919a7754a8c41b6f73199 Nov 02 14:26:25.470065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.470029 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:25.470171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.470098 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:25.470171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.470128 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:26:25.775023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.774905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:25.812599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.812559 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" event={"ID":"bae3377c-0760-40ea-9259-6edc9e44d021","Type":"ContainerStarted","Data":"1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a"} Nov 02 14:26:25.812779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.812604 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg" Nov 02 14:26:25.812779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.812619 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" event={"ID":"bae3377c-0760-40ea-9259-6edc9e44d021","Type":"ContainerStarted","Data":"33b242a8bfa0dc261b2e0e1e7197e0468fba73e9464919a7754a8c41b6f73199"} Nov 02 14:26:25.875360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.875293 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podStartSLOduration=1.875279913 podStartE2EDuration="1.875279913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:25.874122466 +0000 UTC m=+2534.599744237" watchObservedRunningTime="2026-11-02 14:26:25.875279913 +0000 UTC m=+2534.600901684" Nov 02 14:26:25.914393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.914361 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg"] Nov 02 14:26:25.917603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:25.917574 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-7bbcnn2wg"] Nov 02 14:26:26.133872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:26.133846 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:26.135901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:26.135875 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 14:26:26.136029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:26.135927 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 14:26:27.145007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:27.143449 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 14:26:27.145007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:27.143522 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 14:26:27.473540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:27.473443 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1e0ac2-5d9a-410b-8b93-fefb45bd409d" path="/var/lib/kubelet/pods/af1e0ac2-5d9a-410b-8b93-fefb45bd409d/volumes" Nov 02 14:26:27.534019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:27.532471 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:27.882827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:27.882790 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:28.122024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.118610 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:28.145086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.135603 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 14:26:28.145086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.135667 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 14:26:28.483757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.483668 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt"] Nov 02 14:26:28.484741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.484722 5591 cpu_manager.go:414] "containerMap: removing container" podUID="af1e0ac2-5d9a-410b-8b93-fefb45bd409d" containerName="payload-processing" Nov 02 14:26:28.484824 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.484743 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1e0ac2-5d9a-410b-8b93-fefb45bd409d" containerName="payload-processing" Nov 02 14:26:28.484936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.484925 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a1f163-c7e8-480c-b88a-06eaa37bc2b7" containerName="istio-proxy" Nov 02 14:26:28.484979 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.484944 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1e0ac2-5d9a-410b-8b93-fefb45bd409d" containerName="payload-processing" Nov 02 14:26:28.489450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.489430 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.507911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.507879 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt"] Nov 02 14:26:28.517729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.517698 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-821b4dcc9b92\"" Nov 02 14:26:28.517945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.517926 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-e2e--6a9d33b0-dockercfg-b6wf9\"" Nov 02 14:26:28.518040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.518026 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"kube-root-ca.crt\"" Nov 02 14:26:28.518135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.518122 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"routing-overlay\"" Nov 02 14:26:28.523394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.523372 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ee3eca\"" Nov 02 14:26:28.523593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.523577 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"openshift-service-ca.crt\"" Nov 02 14:26:28.526353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.526333 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9vz\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-kube-api-access-tz9vz\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.526584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.526564 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.526669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.526644 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.526818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.526797 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.628857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.628817 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9vz\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-kube-api-access-tz9vz\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.629102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.628953 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.629102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.629025 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.629102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.629093 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.629780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.629757 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.629879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.629846 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.656938 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.656902 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.661576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.660729 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9vz\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-kube-api-access-tz9vz\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:28.803429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:28.803320 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:29.157502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.157465 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 14:26:29.157947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.157532 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 14:26:29.157947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.157644 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt"] Nov 02 14:26:29.160191 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:26:29.160161 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35c6987_88e2_4d84_8bb1_0d8f283fcfc3.slice/crio-f72db2d76f7120fc4ee40397e3234b00e12e3b43f72d0a571221232d847a0cc8 WatchSource:0}: Error finding container f72db2d76f7120fc4ee40397e3234b00e12e3b43f72d0a571221232d847a0cc8: Status 404 returned error can't find the container with id f72db2d76f7120fc4ee40397e3234b00e12e3b43f72d0a571221232d847a0cc8 Nov 02 14:26:29.562066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.558818 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:29.751061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.748402 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:29.870493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.870395 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" event={"ID":"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3","Type":"ContainerStarted","Data":"0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918"} Nov 02 14:26:29.870493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.870450 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" event={"ID":"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3","Type":"ContainerStarted","Data":"f72db2d76f7120fc4ee40397e3234b00e12e3b43f72d0a571221232d847a0cc8"} Nov 02 14:26:29.870493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.870481 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:29.915268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:29.915219 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" podStartSLOduration=1.915204769 podStartE2EDuration="1.915204769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:29.91344434 +0000 UTC m=+2538.639066109" watchObservedRunningTime="2026-11-02 14:26:29.915204769 +0000 UTC m=+2538.640826544" Nov 02 14:26:30.140009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:30.138345 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 14:26:30.140009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:30.138409 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 14:26:31.085536 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:26:31.085500 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:26:31.144005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:31.142368 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 14:26:31.144005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:31.142428 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 14:26:31.175036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:31.174554 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:31.848007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:31.847851 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:32.134226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:32.134196 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 14:26:32.134616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:32.134254 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 14:26:32.895544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:32.895282 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:33.139154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:33.139122 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:33.139769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:33.139740 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:33.140805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:33.140783 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:26:34.879545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:34.879507 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:36.821385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:36.821355 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:37.225566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.225528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:37.415318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.415277 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:37.607992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.607905 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb"] Nov 02 14:26:37.612719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.612697 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.643785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.643748 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"openshift-service-ca.crt\"" Nov 02 14:26:37.644019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.643859 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-e2e--ac2d6d70-dockercfg-gjz62\"" Nov 02 14:26:37.644019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.643959 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"routing-overlay\"" Nov 02 14:26:37.644244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.644040 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a02c35\"" Nov 02 14:26:37.644244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.644130 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"kube-root-ca.crt\"" Nov 02 14:26:37.644244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.644172 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"agc-cred-addf94a36aad\"" Nov 02 14:26:37.647372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.647342 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.647512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.647387 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttfg\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-kube-api-access-2ttfg\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.647641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.647608 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.647711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.647635 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb"] Nov 02 14:26:37.647711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.647674 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.748970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.748930 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.748970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.748975 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttfg\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-kube-api-access-2ttfg\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.749213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.749165 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.749253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.749213 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.749832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.749809 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.749959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.749940 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.751443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.751423 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.777633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.777598 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttfg\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-kube-api-access-2ttfg\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:37.838059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.834953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:37.928713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:37.928675 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:38.012050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:38.012013 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:38.240078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:38.240044 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb"] Nov 02 14:26:38.240363 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:26:38.240337 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed04cbe_9ecc_4c4b_a1a6_8114a894bedb.slice/crio-a149e85c0405318239b535c399c6578c64545007795b69109b84a26976ac300c WatchSource:0}: Error finding container a149e85c0405318239b535c399c6578c64545007795b69109b84a26976ac300c: Status 404 returned error can't find the container with id a149e85c0405318239b535c399c6578c64545007795b69109b84a26976ac300c Nov 02 14:26:38.872443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:38.872405 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:39.029007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:39.019798 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" event={"ID":"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb","Type":"ContainerStarted","Data":"0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc"} Nov 02 14:26:39.029007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:39.019874 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" event={"ID":"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb","Type":"ContainerStarted","Data":"a149e85c0405318239b535c399c6578c64545007795b69109b84a26976ac300c"} Nov 02 14:26:39.029269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:39.029234 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:39.088257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:39.088197 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" podStartSLOduration=2.088183729 podStartE2EDuration="2.088183729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:39.083862994 +0000 UTC m=+2547.809484766" watchObservedRunningTime="2026-11-02 14:26:39.088183729 +0000 UTC m=+2547.813805499" Nov 02 14:26:39.428382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:39.428348 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:40.560273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:40.560238 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:40.880203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:40.880124 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:26:42.289563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:42.289507 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:44.141255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:44.141203 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:44.301628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:44.301575 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:45.738633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:45.738593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:45.929250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:45.929199 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:46.136825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:46.136747 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:47.972345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:47.972281 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:49.654941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:49.654899 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:51.036782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:51.036750 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:26:51.480045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:51.461891 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:53.064526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:53.064495 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:53.602101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:53.602058 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:26:54.761556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:54.761513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:56.489463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:56.488113 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:57.585592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:57.585562 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:57.770335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:57.770301 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:58.080008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:58.077403 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:26:59.740022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:26:59.738716 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:01.074412 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:27:01.074371 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:27:01.611691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:01.611609 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:03.222244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:03.218465 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:04.965738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:04.965690 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:06.650946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:06.650909 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:08.336565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:08.336513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:08.558372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:08.558297 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb"] Nov 02 14:27:08.606581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:08.606495 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:08.772695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:08.772664 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:09.227642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:09.227607 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:10.067614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:10.067570 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:11.699513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:11.694518 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:11.895316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:11.895277 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt"] Nov 02 14:27:11.966618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:11.966529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:12.130813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:12.130770 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:12.638710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:12.638658 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:13.565061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.560064 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" podUID="9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" containerName="payload-processing" containerID="cri-o://0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc" gracePeriod=25 Nov 02 14:27:13.578757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.578712 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:13.847904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.847871 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:27:13.904429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.904399 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-plugins-config-volume\") pod \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " Nov 02 14:27:13.904622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.904448 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-kube-api-access-2ttfg\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-kube-api-access-2ttfg\") pod \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " Nov 02 14:27:13.904622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.904485 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-provider-credentials\") pod \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " Nov 02 14:27:13.904622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.904525 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-routing-overlay-volume\") pod \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\" (UID: \"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb\") " Nov 02 14:27:13.905249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.905062 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-routing-overlay-volume" pod "9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" (UID: "9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:13.905249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.905209 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-plugins-config-volume" pod "9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" (UID: "9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:13.905478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.905345 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:13.905478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.905366 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:13.907670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.907640 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-provider-credentials" pod "9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" (UID: "9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:13.911116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:13.911088 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-kube-api-access-2ttfg" pod "9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" (UID: "9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb"). InnerVolumeSpecName "kube-api-access-2ttfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:14.006470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.006438 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ttfg\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-kube-api-access-2ttfg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:14.006641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.006468 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:14.471025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.470971 5591 generic.go:356] "Generic (PLEG): container finished" podID="9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" containerID="0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc" exitCode=0 Nov 02 14:27:14.471205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.471050 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" Nov 02 14:27:14.471205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.471058 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" event={"ID":"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb","Type":"ContainerDied","Data":"0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc"} Nov 02 14:27:14.471205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.471117 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb" event={"ID":"9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb","Type":"ContainerDied","Data":"a149e85c0405318239b535c399c6578c64545007795b69109b84a26976ac300c"} Nov 02 14:27:14.471205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.471141 5591 scope.go:122] "RemoveContainer" containerID="0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc" Nov 02 14:27:14.488362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.485366 5591 scope.go:122] "RemoveContainer" containerID="0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc" Nov 02 14:27:14.488362 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:27:14.485750 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc\": container with ID starting with 0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc not found: ID does not exist" containerID="0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc" Nov 02 14:27:14.488362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.485788 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc"} err="failed to get container status \"0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc\": rpc error: code = NotFound desc = could not find container \"0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc\": container with ID starting with 0d4e34e02b96f134bee8bd828ee8796464b00479032e61cd3e4524d1c58e3efc not found: ID does not exist" Nov 02 14:27:14.577012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.570050 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb"] Nov 02 14:27:14.595748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:14.595709 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-f9d9hgtdb"] Nov 02 14:27:15.284754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:15.284718 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:15.432821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:15.432784 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" path="/var/lib/kubelet/pods/9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb/volumes" Nov 02 14:27:15.765106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:15.765066 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:15.933209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:15.933178 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:16.895787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:16.895745 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" podUID="c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" containerName="payload-processing" containerID="cri-o://0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918" gracePeriod=25 Nov 02 14:27:17.104090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.103939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:17.226127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.226094 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:27:17.304763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.304725 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-routing-overlay-volume\") pod \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " Nov 02 14:27:17.304957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.304781 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-plugins-config-volume\") pod \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " Nov 02 14:27:17.304957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.304859 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-kube-api-access-tz9vz\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-kube-api-access-tz9vz\") pod \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " Nov 02 14:27:17.304957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.304917 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-provider-credentials\") pod \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\" (UID: \"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3\") " Nov 02 14:27:17.305229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.305201 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-routing-overlay-volume" pod "c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" (UID: "c35c6987-88e2-4d84-8bb1-0d8f283fcfc3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:17.305309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.305251 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-plugins-config-volume" pod "c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" (UID: "c35c6987-88e2-4d84-8bb1-0d8f283fcfc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:17.305934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.305903 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:17.306083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.305937 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:17.307275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.307248 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-kube-api-access-tz9vz" pod "c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" (UID: "c35c6987-88e2-4d84-8bb1-0d8f283fcfc3"). InnerVolumeSpecName "kube-api-access-tz9vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:17.307672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.307647 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-provider-credentials" pod "c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" (UID: "c35c6987-88e2-4d84-8bb1-0d8f283fcfc3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:17.407468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.407433 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz9vz\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-kube-api-access-tz9vz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:17.407468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.407461 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:17.496310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.496230 5591 generic.go:356] "Generic (PLEG): container finished" podID="c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" containerID="0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918" exitCode=0 Nov 02 14:27:17.496310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.496274 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" event={"ID":"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3","Type":"ContainerDied","Data":"0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918"} Nov 02 14:27:17.496310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.496285 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" Nov 02 14:27:17.496579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.496321 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt" event={"ID":"c35c6987-88e2-4d84-8bb1-0d8f283fcfc3","Type":"ContainerDied","Data":"f72db2d76f7120fc4ee40397e3234b00e12e3b43f72d0a571221232d847a0cc8"} Nov 02 14:27:17.496579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.496346 5591 scope.go:122] "RemoveContainer" containerID="0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918" Nov 02 14:27:17.506538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.506516 5591 scope.go:122] "RemoveContainer" containerID="0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918" Nov 02 14:27:17.506816 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:27:17.506795 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918\": container with ID starting with 0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918 not found: ID does not exist" containerID="0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918" Nov 02 14:27:17.506866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.506826 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918"} err="failed to get container status \"0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918\": rpc error: code = NotFound desc = could not find container \"0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918\": container with ID starting with 0962ea1a267ff21aa4cd2e0c73742086a8c84456c6ed92ed91cedc402f7b8918 not found: ID does not exist" Nov 02 14:27:17.552036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.552006 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt"] Nov 02 14:27:17.565334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:17.565306 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5r89bt"] Nov 02 14:27:18.508822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:18.508778 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:18.683338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:18.683301 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:18.854209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:18.854129 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:19.434450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:19.434414 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" path="/var/lib/kubelet/pods/c35c6987-88e2-4d84-8bb1-0d8f283fcfc3/volumes" Nov 02 14:27:20.457911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:20.457869 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:22.050632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:22.050583 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:22.217629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:22.217598 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:22.860899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:22.860862 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:23.120665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:23.120564 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:24.042412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.042371 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:24.238959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.238929 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:24.606010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.605965 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6"] Nov 02 14:27:24.607343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.607304 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" containerName="payload-processing" Nov 02 14:27:24.607343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.607333 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" containerName="payload-processing" Nov 02 14:27:24.607567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.607465 5591 cpu_manager.go:414] "containerMap: removing container" podUID="9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" containerName="payload-processing" Nov 02 14:27:24.607567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.607476 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" containerName="payload-processing" Nov 02 14:27:24.607815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.607795 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35c6987-88e2-4d84-8bb1-0d8f283fcfc3" containerName="payload-processing" Nov 02 14:27:24.607932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.607894 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed04cbe-9ecc-4c4b-a1a6-8114a894bedb" containerName="payload-processing" Nov 02 14:27:24.614387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.614362 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6"] Nov 02 14:27:24.614505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.614468 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.632479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.632455 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"e2e-cmp-legacy-24789c-provider-cred\"" Nov 02 14:27:24.632630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.632508 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5f98a6\"" Nov 02 14:27:24.632798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.632777 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"kube-root-ca.crt\"" Nov 02 14:27:24.632889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.632878 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"openshift-service-ca.crt\"" Nov 02 14:27:24.632927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.632901 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"routing-overlay\"" Nov 02 14:27:24.645166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.645137 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"payload-processing-external-model-e2e--300d9f08-dockercfg-g8r2v\"" Nov 02 14:27:24.799752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.799712 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.799953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.799929 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.800068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.800054 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.800112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.800084 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4xr\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-kube-api-access-lf4xr\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.901109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.901069 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4xr\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-kube-api-access-lf4xr\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.901300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.901203 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.901354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.901337 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.901399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.901386 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.902050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.902025 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.902183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.902113 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.903782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.903757 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.924992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.924954 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4xr\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-kube-api-access-lf4xr\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.929867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.929845 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:24.992160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:24.992131 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:25.239682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:25.239654 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6"] Nov 02 14:27:25.240435 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:27:25.240402 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588a4426_0976_453a_9016_dadbc099c277.slice/crio-4251f16fa84f816398e83b61e96e64377ec5fec6b8dec94a9fc2c0ab6271b6aa WatchSource:0}: Error finding container 4251f16fa84f816398e83b61e96e64377ec5fec6b8dec94a9fc2c0ab6271b6aa: Status 404 returned error can't find the container with id 4251f16fa84f816398e83b61e96e64377ec5fec6b8dec94a9fc2c0ab6271b6aa Nov 02 14:27:25.613117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:25.613019 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" event={"ID":"588a4426-0976-453a-9016-dadbc099c277","Type":"ContainerStarted","Data":"e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c"} Nov 02 14:27:25.613117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:25.613080 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" event={"ID":"588a4426-0976-453a-9016-dadbc099c277","Type":"ContainerStarted","Data":"4251f16fa84f816398e83b61e96e64377ec5fec6b8dec94a9fc2c0ab6271b6aa"} Nov 02 14:27:25.613327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:25.613228 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:25.657881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:25.657825 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" podStartSLOduration=1.6578121019999998 podStartE2EDuration="1.657812102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:25.655530455 +0000 UTC m=+2594.381152228" watchObservedRunningTime="2026-11-02 14:27:25.657812102 +0000 UTC m=+2594.383433872" Nov 02 14:27:25.804007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:25.800601 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:26.039031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:26.038570 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:26.274516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:26.274481 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:26.451108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:26.451081 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:26.696088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:26.694578 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:27.301241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:27.301208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:27.465847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:27.465816 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:28.000604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:28.000568 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:28.518446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:28.518412 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:28.799441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:28.799342 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:28.980944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:28.980902 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv"] Nov 02 14:27:29.052925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.052835 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:29.147520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.147485 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.159618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.159583 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv"] Nov 02 14:27:29.164737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.164708 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"routing-overlay\"" Nov 02 14:27:29.164933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.164909 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"kube-root-ca.crt\"" Nov 02 14:27:29.165065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.164938 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-plugins-e2e-extmod-a-26b508\"" Nov 02 14:27:29.178301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.178259 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"openshift-service-ca.crt\"" Nov 02 14:27:29.178521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.178495 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-1665ff760a19\"" Nov 02 14:27:29.260019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.259958 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbc9s\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-kube-api-access-dbc9s\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.260725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.260605 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.261266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.261133 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.265003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.261550 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.368063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.367948 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.368423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.368392 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.369124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.369100 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.369437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.369419 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbc9s\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-kube-api-access-dbc9s\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.370798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.370772 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.371705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.371627 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.401659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.401625 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.401936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.401885 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbc9s\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-kube-api-access-dbc9s\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.469064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.469028 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:29.784470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.784438 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln"] Nov 02 14:27:29.788045 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:27:29.788012 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f187c9_0d67_4260_96ec_ead86fbd8dfd.slice/crio-dde37f89d6f9606ec26acce9035a11884c1bf7cb87ad7d089cc8e5b855b79360 WatchSource:0}: Error finding container dde37f89d6f9606ec26acce9035a11884c1bf7cb87ad7d089cc8e5b855b79360: Status 404 returned error can't find the container with id dde37f89d6f9606ec26acce9035a11884c1bf7cb87ad7d089cc8e5b855b79360 Nov 02 14:27:29.791250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.791228 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv"] Nov 02 14:27:29.791362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.791343 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.808692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.808646 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-e2e--7e93d834-dockercfg-vv77g\"" Nov 02 14:27:29.808857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.808688 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln"] Nov 02 14:27:29.876728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.876698 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.876971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.876943 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.877042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.876999 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.877115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.877097 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894x5\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-kube-api-access-894x5\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.908832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.908804 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-85e62aa25307\"" Nov 02 14:27:29.983060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.978537 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894x5\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-kube-api-access-894x5\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.983060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.978695 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.983060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.978943 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.983060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.979000 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.985267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.985184 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:29.991773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.990656 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.991773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.991291 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:29.995236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:29.995204 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:30.008903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.008867 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894x5\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-kube-api-access-894x5\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:30.118105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.117977 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:30.425196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.425166 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln"] Nov 02 14:27:30.425439 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:27:30.425410 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02821c8b_2af5_47f6_b552_ea28274991b3.slice/crio-9d8284b690557b8c8ff1212481e7fce376b58cb7e60d8c5643abe7ce9fb3479a WatchSource:0}: Error finding container 9d8284b690557b8c8ff1212481e7fce376b58cb7e60d8c5643abe7ce9fb3479a: Status 404 returned error can't find the container with id 9d8284b690557b8c8ff1212481e7fce376b58cb7e60d8c5643abe7ce9fb3479a Nov 02 14:27:30.716465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.716379 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" event={"ID":"02821c8b-2af5-47f6-b552-ea28274991b3","Type":"ContainerStarted","Data":"bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4"} Nov 02 14:27:30.716465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.716436 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" event={"ID":"02821c8b-2af5-47f6-b552-ea28274991b3","Type":"ContainerStarted","Data":"9d8284b690557b8c8ff1212481e7fce376b58cb7e60d8c5643abe7ce9fb3479a"} Nov 02 14:27:30.716723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.716578 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:30.718198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.718172 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" event={"ID":"d9f187c9-0d67-4260-96ec-ead86fbd8dfd","Type":"ContainerStarted","Data":"aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2"} Nov 02 14:27:30.718327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.718207 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" event={"ID":"d9f187c9-0d67-4260-96ec-ead86fbd8dfd","Type":"ContainerStarted","Data":"dde37f89d6f9606ec26acce9035a11884c1bf7cb87ad7d089cc8e5b855b79360"} Nov 02 14:27:30.718327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.718228 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:30.760304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.760256 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" podStartSLOduration=1.7602424810000001 podStartE2EDuration="1.760242481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:30.758744001 +0000 UTC m=+2599.484365753" watchObservedRunningTime="2026-11-02 14:27:30.760242481 +0000 UTC m=+2599.485864250" Nov 02 14:27:30.802324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:30.802275 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" podStartSLOduration=2.802262564 podStartE2EDuration="2.802262564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:30.8018778 +0000 UTC m=+2599.527499571" watchObservedRunningTime="2026-11-02 14:27:30.802262564 +0000 UTC m=+2599.527884374" Nov 02 14:27:31.075016 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:27:31.074904 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:27:31.588668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:31.588629 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:33.338633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:33.338599 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:35.092770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:35.083151 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:35.679797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:35.679756 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:36.623549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:36.623519 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:27:36.755179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:36.755146 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:36.929867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:36.929825 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:37.113318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:37.113277 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:37.450561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:37.450488 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:37.645012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:37.644942 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:38.136373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:38.136339 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:27:38.275674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:38.275643 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:38.472819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:38.472727 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:38.844379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:38.844283 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:40.241682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.241646 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:40.443272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.443237 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r"] Nov 02 14:27:40.447883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.447862 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.468935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.468904 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r"] Nov 02 14:27:40.477433 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.477395 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"agc-cred-9b11dc31ee6b\"" Nov 02 14:27:40.477695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.477672 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"routing-overlay\"" Nov 02 14:27:40.477815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.477798 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"kube-root-ca.crt\"" Nov 02 14:27:40.477938 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.477919 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"payload-processing-external-model-e2e--47c96fd3-dockercfg-6h7wl\"" Nov 02 14:27:40.479306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.479283 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"openshift-service-ca.crt\"" Nov 02 14:27:40.479415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.479362 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-66bc6d\"" Nov 02 14:27:40.541783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.541698 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.541783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.541774 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.541973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.541892 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbjv\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-kube-api-access-6mbjv\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.542247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.542228 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.655009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.644437 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.655009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.644528 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.655009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.644618 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.655009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.644665 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbjv\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-kube-api-access-6mbjv\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.655009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.646059 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.664004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.661741 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.664004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.662325 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.683010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.678694 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbjv\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-kube-api-access-6mbjv\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.763543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.762280 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:40.883632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:40.882753 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:41.135686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:41.131407 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r"] Nov 02 14:27:41.729507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:41.729480 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:27:41.730135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:41.729557 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:41.823042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:41.823006 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv"] Nov 02 14:27:41.827273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:41.827233 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" event={"ID":"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8","Type":"ContainerStarted","Data":"81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32"} Nov 02 14:27:41.827447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:41.827301 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" event={"ID":"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8","Type":"ContainerStarted","Data":"bfe848248acf68ffc2f63156e9a02e6ed2a2490791b74325cd6b0ba603661a4e"} Nov 02 14:27:41.827447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:41.827331 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:41.929501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:41.929431 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" podStartSLOduration=1.929410189 podStartE2EDuration="1.929410189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:41.924207535 +0000 UTC m=+2610.649829307" watchObservedRunningTime="2026-11-02 14:27:41.929410189 +0000 UTC m=+2610.655031951" Nov 02 14:27:42.677843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:42.677803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:44.378548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:44.378510 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:45.960966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:45.960924 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:46.828011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:46.827930 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" podUID="d9f187c9-0d67-4260-96ec-ead86fbd8dfd" containerName="payload-processing" containerID="cri-o://aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2" gracePeriod=25 Nov 02 14:27:47.108683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.108653 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:47.146390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.146357 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-routing-overlay-volume\") pod \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " Nov 02 14:27:47.146564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.146548 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-kube-api-access-dbc9s\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-kube-api-access-dbc9s\") pod \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " Nov 02 14:27:47.146660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.146642 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-plugins-config-volume\") pod \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " Nov 02 14:27:47.146781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.146760 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-routing-overlay-volume" pod "d9f187c9-0d67-4260-96ec-ead86fbd8dfd" (UID: "d9f187c9-0d67-4260-96ec-ead86fbd8dfd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:47.146847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.146784 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-provider-credentials\") pod \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\" (UID: \"d9f187c9-0d67-4260-96ec-ead86fbd8dfd\") " Nov 02 14:27:47.147077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.147046 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-plugins-config-volume" pod "d9f187c9-0d67-4260-96ec-ead86fbd8dfd" (UID: "d9f187c9-0d67-4260-96ec-ead86fbd8dfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:47.147436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.147413 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:47.147519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.147440 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:47.148910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.148882 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-kube-api-access-dbc9s" pod "d9f187c9-0d67-4260-96ec-ead86fbd8dfd" (UID: "d9f187c9-0d67-4260-96ec-ead86fbd8dfd"). InnerVolumeSpecName "kube-api-access-dbc9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:47.149057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.148918 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-provider-credentials" pod "d9f187c9-0d67-4260-96ec-ead86fbd8dfd" (UID: "d9f187c9-0d67-4260-96ec-ead86fbd8dfd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:47.248970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.248929 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:47.248970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.248969 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbc9s\" (UniqueName: \"kubernetes.io/projected/d9f187c9-0d67-4260-96ec-ead86fbd8dfd-kube-api-access-dbc9s\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:27:47.663300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.663206 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:47.872054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.871955 5591 generic.go:356] "Generic (PLEG): container finished" podID="d9f187c9-0d67-4260-96ec-ead86fbd8dfd" containerID="aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2" exitCode=0 Nov 02 14:27:47.872054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.872038 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" Nov 02 14:27:47.872264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.872047 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" event={"ID":"d9f187c9-0d67-4260-96ec-ead86fbd8dfd","Type":"ContainerDied","Data":"aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2"} Nov 02 14:27:47.872264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.872108 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv" event={"ID":"d9f187c9-0d67-4260-96ec-ead86fbd8dfd","Type":"ContainerDied","Data":"dde37f89d6f9606ec26acce9035a11884c1bf7cb87ad7d089cc8e5b855b79360"} Nov 02 14:27:47.872264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.872140 5591 scope.go:122] "RemoveContainer" containerID="aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2" Nov 02 14:27:47.882724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.882704 5591 scope.go:122] "RemoveContainer" containerID="aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2" Nov 02 14:27:47.883028 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:27:47.883008 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2\": container with ID starting with aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2 not found: ID does not exist" containerID="aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2" Nov 02 14:27:47.883094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.883035 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2"} err="failed to get container status \"aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2\": rpc error: code = NotFound desc = could not find container \"aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2\": container with ID starting with aa7dd708fd76e8ecec09631c863dea0a8e77a951383ad6868c26d47fd987e9d2 not found: ID does not exist" Nov 02 14:27:47.929007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.928947 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv"] Nov 02 14:27:47.941704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:47.941672 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96qhcnv"] Nov 02 14:27:49.404006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:49.401927 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:49.433367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:49.433329 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f187c9-0d67-4260-96ec-ead86fbd8dfd" path="/var/lib/kubelet/pods/d9f187c9-0d67-4260-96ec-ead86fbd8dfd/volumes" Nov 02 14:27:51.172553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:51.172513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:52.774068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:52.773692 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:52.837483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:52.837452 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:27:53.902240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:53.902198 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:54.088140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:54.088104 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:54.823200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:54.823160 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:55.079073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.078181 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:55.282417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.282375 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m"] Nov 02 14:27:55.283143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.283127 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d9f187c9-0d67-4260-96ec-ead86fbd8dfd" containerName="payload-processing" Nov 02 14:27:55.283205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.283144 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f187c9-0d67-4260-96ec-ead86fbd8dfd" containerName="payload-processing" Nov 02 14:27:55.283313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.283304 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f187c9-0d67-4260-96ec-ead86fbd8dfd" containerName="payload-processing" Nov 02 14:27:55.288436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.288412 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.307646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.307617 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m"] Nov 02 14:27:55.316995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.316955 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"agc-cred-187d768557d1\"" Nov 02 14:27:55.317160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.316963 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"payload-processing-external-model-e2e--1d26709d-dockercfg-v6xgj\"" Nov 02 14:27:55.317160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.317077 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"e2e-cfl-4239ef-provider-cred\"" Nov 02 14:27:55.317160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.317123 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"openshift-service-ca.crt\"" Nov 02 14:27:55.317332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.317173 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"routing-overlay\"" Nov 02 14:27:55.317399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.317393 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"kube-root-ca.crt\"" Nov 02 14:27:55.317460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.317422 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2d13f4\"" Nov 02 14:27:55.382626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.382584 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.382626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.382676 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4jv\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-kube-api-access-bs4jv\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.382961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.382879 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.382961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.382939 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.484353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.484313 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.484539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.484390 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.484539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.484517 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.484651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.484574 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4jv\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-kube-api-access-bs4jv\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.485222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.485196 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.485359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.485230 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.487122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.487095 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.509664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.509638 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4jv\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-kube-api-access-bs4jv\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.603319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.603285 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:55.977338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:55.977250 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m"] Nov 02 14:27:56.671589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:56.669197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:27:56.986547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:56.986321 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" event={"ID":"6cc66e73-57d8-46d0-8159-121a3df2b282","Type":"ContainerStarted","Data":"60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1"} Nov 02 14:27:56.986547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:56.986424 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" event={"ID":"6cc66e73-57d8-46d0-8159-121a3df2b282","Type":"ContainerStarted","Data":"090071f374b767bbd83beda4044759d9bf7f57e508efe556cc57a7e1a57e1e34"} Nov 02 14:27:56.986547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:56.986496 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:27:57.033554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:57.033501 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" podStartSLOduration=2.03348861 podStartE2EDuration="2.03348861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:57.030167757 +0000 UTC m=+2625.755789537" watchObservedRunningTime="2026-11-02 14:27:57.03348861 +0000 UTC m=+2625.759110379" Nov 02 14:27:58.565008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:27:58.559574 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:00.178010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:00.177651 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:00.347229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:00.347194 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:01.076675 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:28:01.076627 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:28:01.785719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:01.779068 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:03.461817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:03.461774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:05.057543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:05.057504 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:06.848869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:06.848835 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:07.025906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:07.025854 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:07.996970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:07.996938 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:28:08.635273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:08.635218 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:10.247802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:10.247762 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:11.972695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:11.966864 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:13.674150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:13.674111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:15.422693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:15.422658 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:17.094007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:17.086555 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:18.732641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:18.732596 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:19.038479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:19.038380 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:19.222908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:19.222869 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:19.671625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:19.671591 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:20.535702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:20.534263 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:22.154611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:22.154569 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:23.851843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:23.847521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:25.117304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:25.117246 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:25.469863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:25.469815 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:26.166673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:26.166591 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:26.370730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:26.370699 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:27.161020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:27.160349 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:28.905636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:28.905603 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:30.672623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:30.672568 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:31.075713 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:28:31.075627 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:28:31.854655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:31.854622 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:28:32.398404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:32.398222 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:32.914675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:32.914636 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:33.109513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:33.109477 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:33.415255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:33.415211 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:33.658170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:33.658079 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:34.459951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.459910 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:34.644969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.644903 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:34.734633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.734558 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9"] Nov 02 14:28:34.739654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.739628 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:34.761608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.761579 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9"] Nov 02 14:28:34.769728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.769697 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"openshift-service-ca.crt\"" Nov 02 14:28:34.769916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.769705 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"routing-overlay\"" Nov 02 14:28:34.769916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.769790 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"kube-root-ca.crt\"" Nov 02 14:28:34.769916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.769820 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-87efd3\"" Nov 02 14:28:34.772684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.772661 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"payload-processing-external-model-e2e--005c2a0a-dockercfg-wl4vh\"" Nov 02 14:28:34.772684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.772676 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"e2e-cmp-legacy-f1c3a9-provider-cred\"" Nov 02 14:28:34.921059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.920949 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:34.921400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.921377 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnp8l\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-kube-api-access-jnp8l\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:34.921556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.921537 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:34.926007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:34.921882 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.031358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.031252 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.033849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.033820 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnp8l\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-kube-api-access-jnp8l\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.034466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.034428 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.034927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.034902 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.041747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.032643 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.041747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.035885 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.053054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.043706 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.061347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.061317 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnp8l\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-kube-api-access-jnp8l\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.355364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.355268 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:35.677827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:35.677800 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9"] Nov 02 14:28:35.678383 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:28:35.678358 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c48aaf7_0bed_4583_87c3_5f1af8cf98ce.slice/crio-96472b0fd00411f9771adf8adc9fdeae764c2d29e578a9b2ace4f06c13de5365 WatchSource:0}: Error finding container 96472b0fd00411f9771adf8adc9fdeae764c2d29e578a9b2ace4f06c13de5365: Status 404 returned error can't find the container with id 96472b0fd00411f9771adf8adc9fdeae764c2d29e578a9b2ace4f06c13de5365 Nov 02 14:28:36.304366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:36.304330 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" event={"ID":"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce","Type":"ContainerStarted","Data":"96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a"} Nov 02 14:28:36.304552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:36.304381 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" event={"ID":"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce","Type":"ContainerStarted","Data":"96472b0fd00411f9771adf8adc9fdeae764c2d29e578a9b2ace4f06c13de5365"} Nov 02 14:28:36.304552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:36.304416 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:36.351583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:36.351531 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" podStartSLOduration=2.351517472 podStartE2EDuration="2.351517472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:36.349056186 +0000 UTC m=+2665.074677955" watchObservedRunningTime="2026-11-02 14:28:36.351517472 +0000 UTC m=+2665.077139241" Nov 02 14:28:36.426340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:36.426307 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:37.475087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:37.475053 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:37.657570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:37.657521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:38.415843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.403431 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:38.842020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.841931 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44"] Nov 02 14:28:38.846719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.846697 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:38.865384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.865350 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44"] Nov 02 14:28:38.874123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.874092 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"agc-cred-dab42deaef05\"" Nov 02 14:28:38.907032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.906977 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:38.907186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.907076 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:38.907186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.907173 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:38.907396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:38.907370 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dtvm\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-kube-api-access-6dtvm\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.008623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.008581 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.008815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.008631 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.008815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.008735 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.008940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.008915 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtvm\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-kube-api-access-6dtvm\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.009383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.009351 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.009518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.009434 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.011350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.011319 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.033307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.033279 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtvm\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-kube-api-access-6dtvm\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.162951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.162912 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:39.471716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:39.471684 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44"] Nov 02 14:28:39.472174 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:28:39.472145 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbbf5c4_8864_4a37_b772_a62538a5e622.slice/crio-8b1ff1e1ec7ef53ac0abb1610eff282fb2fa4111c2a1683043d221383906ee12 WatchSource:0}: Error finding container 8b1ff1e1ec7ef53ac0abb1610eff282fb2fa4111c2a1683043d221383906ee12: Status 404 returned error can't find the container with id 8b1ff1e1ec7ef53ac0abb1610eff282fb2fa4111c2a1683043d221383906ee12 Nov 02 14:28:40.192007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:40.191926 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:40.337647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:40.337600 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" event={"ID":"7dbbf5c4-8864-4a37-b772-a62538a5e622","Type":"ContainerStarted","Data":"5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55"} Nov 02 14:28:40.337822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:40.337663 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:40.337822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:40.337684 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" event={"ID":"7dbbf5c4-8864-4a37-b772-a62538a5e622","Type":"ContainerStarted","Data":"8b1ff1e1ec7ef53ac0abb1610eff282fb2fa4111c2a1683043d221383906ee12"} Nov 02 14:28:40.383881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:40.383826 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" podStartSLOduration=2.383809185 podStartE2EDuration="2.383809185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:40.380124013 +0000 UTC m=+2669.105745784" watchObservedRunningTime="2026-11-02 14:28:40.383809185 +0000 UTC m=+2669.109430955" Nov 02 14:28:41.864061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:41.863916 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:43.441649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:43.441611 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:45.150605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:45.150560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:46.795638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:46.795596 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:47.313721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:47.313681 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:28:48.490954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:48.490905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:50.166352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:50.166304 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:50.343819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:50.343780 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:28:51.366417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:51.366384 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:28:52.009006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:52.008830 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:52.046674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:52.046638 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m"] Nov 02 14:28:53.782325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:53.782288 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:55.138308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:55.138271 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:55.302683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:55.302645 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:55.768702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:55.768666 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:57.058900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.058854 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" podUID="6cc66e73-57d8-46d0-8159-121a3df2b282" containerName="payload-processing" containerID="cri-o://60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1" gracePeriod=25 Nov 02 14:28:57.338160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.338128 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:28:57.398516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.398479 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-provider-credentials\") pod \"6cc66e73-57d8-46d0-8159-121a3df2b282\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " Nov 02 14:28:57.398738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.398634 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-plugins-config-volume\") pod \"6cc66e73-57d8-46d0-8159-121a3df2b282\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " Nov 02 14:28:57.398808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.398736 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-kube-api-access-bs4jv\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-kube-api-access-bs4jv\") pod \"6cc66e73-57d8-46d0-8159-121a3df2b282\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " Nov 02 14:28:57.398956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.398932 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-routing-overlay-volume\") pod \"6cc66e73-57d8-46d0-8159-121a3df2b282\" (UID: \"6cc66e73-57d8-46d0-8159-121a3df2b282\") " Nov 02 14:28:57.399326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.399287 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-plugins-config-volume" pod "6cc66e73-57d8-46d0-8159-121a3df2b282" (UID: "6cc66e73-57d8-46d0-8159-121a3df2b282"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:57.399437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.399327 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-routing-overlay-volume" pod "6cc66e73-57d8-46d0-8159-121a3df2b282" (UID: "6cc66e73-57d8-46d0-8159-121a3df2b282"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:57.399857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.399829 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:28:57.399925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.399863 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cc66e73-57d8-46d0-8159-121a3df2b282-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:28:57.401069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.401038 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-provider-credentials" pod "6cc66e73-57d8-46d0-8159-121a3df2b282" (UID: "6cc66e73-57d8-46d0-8159-121a3df2b282"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:57.402411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.402385 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-kube-api-access-bs4jv" pod "6cc66e73-57d8-46d0-8159-121a3df2b282" (UID: "6cc66e73-57d8-46d0-8159-121a3df2b282"). InnerVolumeSpecName "kube-api-access-bs4jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:57.501571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.501526 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:28:57.501571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.501562 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4jv\" (UniqueName: \"kubernetes.io/projected/6cc66e73-57d8-46d0-8159-121a3df2b282-kube-api-access-bs4jv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:28:57.539246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.539208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:57.768011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.767949 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:57.930008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.929735 5591 generic.go:356] "Generic (PLEG): container finished" podID="6cc66e73-57d8-46d0-8159-121a3df2b282" containerID="60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1" exitCode=0 Nov 02 14:28:57.930008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.929786 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" event={"ID":"6cc66e73-57d8-46d0-8159-121a3df2b282","Type":"ContainerDied","Data":"60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1"} Nov 02 14:28:57.930008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.929840 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" event={"ID":"6cc66e73-57d8-46d0-8159-121a3df2b282","Type":"ContainerDied","Data":"090071f374b767bbd83beda4044759d9bf7f57e508efe556cc57a7e1a57e1e34"} Nov 02 14:28:57.930008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.929872 5591 scope.go:122] "RemoveContainer" containerID="60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1" Nov 02 14:28:57.935004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:57.930476 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m" Nov 02 14:28:58.029588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:58.029558 5591 scope.go:122] "RemoveContainer" containerID="60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1" Nov 02 14:28:58.032099 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:28:58.032062 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1\": container with ID starting with 60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1 not found: ID does not exist" containerID="60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1" Nov 02 14:28:58.032308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:58.032280 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1"} err="failed to get container status \"60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1\": rpc error: code = NotFound desc = could not find container \"60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1\": container with ID starting with 60226b9a6155b521f5b5ba573fd92e773a74f71bff5b32b1b6325e7fbec6e8b1 not found: ID does not exist" Nov 02 14:28:58.059752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:58.059719 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m"] Nov 02 14:28:58.073643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:58.073606 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d6svq4m"] Nov 02 14:28:58.917383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:58.917351 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:59.331736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:59.331634 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:28:59.440622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:28:59.440582 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc66e73-57d8-46d0-8159-121a3df2b282" path="/var/lib/kubelet/pods/6cc66e73-57d8-46d0-8159-121a3df2b282/volumes" Nov 02 14:29:00.868970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:00.868934 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:01.043908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:01.043859 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:01.075097 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:29:01.075057 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:29:02.548075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:02.548042 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:02.742525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:02.742483 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:03.023476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:03.023437 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:04.821113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:04.821077 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:04.994735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:04.994692 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:06.569474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:06.569436 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:08.274708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:08.274634 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:09.900691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:09.900642 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:11.620304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:11.620255 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:12.017130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:12.017101 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:29:12.047640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:12.047613 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:29:13.274876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:13.274829 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:13.553042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:13.552960 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6"] Nov 02 14:29:13.692734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:13.692702 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:13.882098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:13.882062 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:14.057446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:14.057410 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:14.409420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:14.409388 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:14.603970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:14.603935 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:14.821754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:14.821669 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:15.020130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:15.020097 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:15.226008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:15.225746 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:15.822613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:15.822578 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:15.987722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:15.987687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:16.376037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:16.375948 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:16.543847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:16.543812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:17.004991 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:17.004946 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:17.201781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:17.201744 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:17.611209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:17.611171 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:18.554197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.554121 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" podUID="588a4426-0976-453a-9016-dadbc099c277" containerName="payload-processing" containerID="cri-o://e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c" gracePeriod=25 Nov 02 14:29:18.828488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.828458 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:29:18.938854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.938824 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-kube-api-access-lf4xr\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-kube-api-access-lf4xr\") pod \"588a4426-0976-453a-9016-dadbc099c277\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " Nov 02 14:29:18.939058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.938882 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-provider-credentials\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-provider-credentials\") pod \"588a4426-0976-453a-9016-dadbc099c277\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " Nov 02 14:29:18.939058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.938907 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-plugins-config-volume\") pod \"588a4426-0976-453a-9016-dadbc099c277\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " Nov 02 14:29:18.939058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.939019 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-routing-overlay-volume\") pod \"588a4426-0976-453a-9016-dadbc099c277\" (UID: \"588a4426-0976-453a-9016-dadbc099c277\") " Nov 02 14:29:18.939450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.939419 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-plugins-config-volume" pod "588a4426-0976-453a-9016-dadbc099c277" (UID: "588a4426-0976-453a-9016-dadbc099c277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:18.939584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.939439 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-routing-overlay-volume" pod "588a4426-0976-453a-9016-dadbc099c277" (UID: "588a4426-0976-453a-9016-dadbc099c277"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:18.939735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.939714 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:18.939800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.939741 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/588a4426-0976-453a-9016-dadbc099c277-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:18.941160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.941136 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-kube-api-access-lf4xr" pod "588a4426-0976-453a-9016-dadbc099c277" (UID: "588a4426-0976-453a-9016-dadbc099c277"). InnerVolumeSpecName "kube-api-access-lf4xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:18.941324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.941306 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-provider-credentials" pod "588a4426-0976-453a-9016-dadbc099c277" (UID: "588a4426-0976-453a-9016-dadbc099c277"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:18.977108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:18.977073 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:19.041336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.041301 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf4xr\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-kube-api-access-lf4xr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:19.041336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.041332 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/588a4426-0976-453a-9016-dadbc099c277-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:19.210040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.209999 5591 generic.go:356] "Generic (PLEG): container finished" podID="588a4426-0976-453a-9016-dadbc099c277" containerID="e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c" exitCode=0 Nov 02 14:29:19.210255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.210045 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" Nov 02 14:29:19.210255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.210079 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" event={"ID":"588a4426-0976-453a-9016-dadbc099c277","Type":"ContainerDied","Data":"e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c"} Nov 02 14:29:19.210255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.210149 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6" event={"ID":"588a4426-0976-453a-9016-dadbc099c277","Type":"ContainerDied","Data":"4251f16fa84f816398e83b61e96e64377ec5fec6b8dec94a9fc2c0ab6271b6aa"} Nov 02 14:29:19.210255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.210180 5591 scope.go:122] "RemoveContainer" containerID="e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c" Nov 02 14:29:19.221206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.221186 5591 scope.go:122] "RemoveContainer" containerID="e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c" Nov 02 14:29:19.221455 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:29:19.221437 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c\": container with ID starting with e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c not found: ID does not exist" containerID="e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c" Nov 02 14:29:19.221505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.221465 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c"} err="failed to get container status \"e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c\": rpc error: code = NotFound desc = could not find container \"e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c\": container with ID starting with e3d639bf6f658dc3d01c673a372ba408106f96942443e5170b02fc4e9320689c not found: ID does not exist" Nov 02 14:29:19.265826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.265793 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6"] Nov 02 14:29:19.269291 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.269267 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-c768wrkm6"] Nov 02 14:29:19.432015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:19.431946 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588a4426-0976-453a-9016-dadbc099c277" path="/var/lib/kubelet/pods/588a4426-0976-453a-9016-dadbc099c277/volumes" Nov 02 14:29:20.690820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:20.690776 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:22.291247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:22.287614 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:24.051660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:24.051561 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:25.235685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:25.235650 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:25.659630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:25.659574 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:27.243090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:27.243052 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:29:27.284494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:27.278387 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:29.221817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:29.221762 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:29.309832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:29.309788 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r"] Nov 02 14:29:29.391611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:29.391561 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:29.555435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:29.555326 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:30.035157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:30.035128 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:31.076397 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:29:31.076355 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:29:31.163206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:31.163173 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:32.167463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:32.163775 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:32.357119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:32.357084 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:33.163007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.157888 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:33.308546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.308514 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l"] Nov 02 14:29:33.309294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.309278 5591 cpu_manager.go:414] "containerMap: removing container" podUID="588a4426-0976-453a-9016-dadbc099c277" containerName="payload-processing" Nov 02 14:29:33.309294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.309295 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588a4426-0976-453a-9016-dadbc099c277" containerName="payload-processing" Nov 02 14:29:33.309398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.309347 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6cc66e73-57d8-46d0-8159-121a3df2b282" containerName="payload-processing" Nov 02 14:29:33.309398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.309354 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc66e73-57d8-46d0-8159-121a3df2b282" containerName="payload-processing" Nov 02 14:29:33.309484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.309474 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc66e73-57d8-46d0-8159-121a3df2b282" containerName="payload-processing" Nov 02 14:29:33.309527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.309487 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="588a4426-0976-453a-9016-dadbc099c277" containerName="payload-processing" Nov 02 14:29:33.314573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.314552 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.335366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.335328 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l"] Nov 02 14:29:33.344721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.344690 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-e1e8442d7168\"" Nov 02 14:29:33.354026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.353974 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.354271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.354241 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hh4\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-kube-api-access-v4hh4\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.354486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.354466 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.354561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.354511 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.455443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.455343 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hh4\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-kube-api-access-v4hh4\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.455737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.455497 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.455737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.455526 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.455737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.455599 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.456444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.456418 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.456549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.456476 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.458519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.458494 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.489777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.489709 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hh4\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-kube-api-access-v4hh4\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.629393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.629340 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:33.960763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:33.960732 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l"] Nov 02 14:29:33.961144 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:29:33.961097 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9408e7_09ae_4c84_82fc_58ec9f46f231.slice/crio-23d2ba5225eaf78987b74ff5175707159e5231895a863e06492b93ebf61cf064 WatchSource:0}: Error finding container 23d2ba5225eaf78987b74ff5175707159e5231895a863e06492b93ebf61cf064: Status 404 returned error can't find the container with id 23d2ba5225eaf78987b74ff5175707159e5231895a863e06492b93ebf61cf064 Nov 02 14:29:34.310634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.310517 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" podUID="e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" containerName="payload-processing" containerID="cri-o://81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32" gracePeriod=25 Nov 02 14:29:34.355585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.355551 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" event={"ID":"4d9408e7-09ae-4c84-82fc-58ec9f46f231","Type":"ContainerStarted","Data":"e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312"} Nov 02 14:29:34.355737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.355608 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" event={"ID":"4d9408e7-09ae-4c84-82fc-58ec9f46f231","Type":"ContainerStarted","Data":"23d2ba5225eaf78987b74ff5175707159e5231895a863e06492b93ebf61cf064"} Nov 02 14:29:34.355737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.355634 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:34.403916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.403863 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" podStartSLOduration=1.403849429 podStartE2EDuration="1.403849429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:34.400735838 +0000 UTC m=+2723.126357608" watchObservedRunningTime="2026-11-02 14:29:34.403849429 +0000 UTC m=+2723.129471225" Nov 02 14:29:34.592603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.592577 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:29:34.671827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.671784 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-kube-api-access-6mbjv\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-kube-api-access-6mbjv\") pod \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " Nov 02 14:29:34.672040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.671880 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-plugins-config-volume\") pod \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " Nov 02 14:29:34.672432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.672399 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-plugins-config-volume" pod "e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" (UID: "e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:34.672574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.672523 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-provider-credentials\") pod \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " Nov 02 14:29:34.673027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.673003 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-routing-overlay-volume\") pod \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\" (UID: \"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8\") " Nov 02 14:29:34.673773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.673740 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-routing-overlay-volume" pod "e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" (UID: "e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:34.674237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.674211 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:34.674237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.674236 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:34.674862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.674835 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-kube-api-access-6mbjv" pod "e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" (UID: "e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8"). InnerVolumeSpecName "kube-api-access-6mbjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:34.675976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.675896 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-provider-credentials" pod "e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" (UID: "e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:34.775746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.775715 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mbjv\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-kube-api-access-6mbjv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:34.775746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.775743 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:34.897660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:34.896747 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:35.365591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.365503 5591 generic.go:356] "Generic (PLEG): container finished" podID="e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" containerID="81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32" exitCode=0 Nov 02 14:29:35.365591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.365551 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" Nov 02 14:29:35.366098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.365589 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" event={"ID":"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8","Type":"ContainerDied","Data":"81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32"} Nov 02 14:29:35.366098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.365652 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r" event={"ID":"e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8","Type":"ContainerDied","Data":"bfe848248acf68ffc2f63156e9a02e6ed2a2490791b74325cd6b0ba603661a4e"} Nov 02 14:29:35.366098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.365677 5591 scope.go:122] "RemoveContainer" containerID="81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32" Nov 02 14:29:35.376514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.376494 5591 scope.go:122] "RemoveContainer" containerID="81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32" Nov 02 14:29:35.376791 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:29:35.376768 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32\": container with ID starting with 81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32 not found: ID does not exist" containerID="81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32" Nov 02 14:29:35.376861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.376806 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32"} err="failed to get container status \"81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32\": rpc error: code = NotFound desc = could not find container \"81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32\": container with ID starting with 81b6446c4dc5a545a20dd18747fc3af2c4045bbadb2042af36e10ad589556e32 not found: ID does not exist" Nov 02 14:29:35.435288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.435253 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r"] Nov 02 14:29:35.436408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:35.436380 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-78ffj7c4r"] Nov 02 14:29:36.493631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:36.493599 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:36.658286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:36.658229 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:36.823716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:36.823632 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:37.433422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:37.433388 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" path="/var/lib/kubelet/pods/e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8/volumes" Nov 02 14:29:38.526322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:38.526282 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:40.186853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:40.186024 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:41.062654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:41.062621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:41.790521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:41.790467 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:43.593673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:43.593634 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:43.852338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:43.852213 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:44.049949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:44.049919 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:44.254466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:44.245772 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:45.371647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:45.371600 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:45.383598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:45.383567 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:29:45.476545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:45.476506 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln"] Nov 02 14:29:47.092006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:47.091202 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:48.764387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:48.761541 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:50.332967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.332922 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:50.476965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.476929 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" podUID="02821c8b-2af5-47f6-b552-ea28274991b3" containerName="payload-processing" containerID="cri-o://bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4" gracePeriod=25 Nov 02 14:29:50.762847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.762817 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:29:50.860599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.860521 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-provider-credentials\") pod \"02821c8b-2af5-47f6-b552-ea28274991b3\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " Nov 02 14:29:50.860599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.860583 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-kube-api-access-894x5\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-kube-api-access-894x5\") pod \"02821c8b-2af5-47f6-b552-ea28274991b3\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " Nov 02 14:29:50.860792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.860647 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-routing-overlay-volume\") pod \"02821c8b-2af5-47f6-b552-ea28274991b3\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " Nov 02 14:29:50.860792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.860674 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-plugins-config-volume\") pod \"02821c8b-2af5-47f6-b552-ea28274991b3\" (UID: \"02821c8b-2af5-47f6-b552-ea28274991b3\") " Nov 02 14:29:50.861058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.861028 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-routing-overlay-volume" pod "02821c8b-2af5-47f6-b552-ea28274991b3" (UID: "02821c8b-2af5-47f6-b552-ea28274991b3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:50.861193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.861168 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-plugins-config-volume" pod "02821c8b-2af5-47f6-b552-ea28274991b3" (UID: "02821c8b-2af5-47f6-b552-ea28274991b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:50.861459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.861443 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:50.861509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.861460 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02821c8b-2af5-47f6-b552-ea28274991b3-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:50.862817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.862791 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-provider-credentials" pod "02821c8b-2af5-47f6-b552-ea28274991b3" (UID: "02821c8b-2af5-47f6-b552-ea28274991b3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:50.862892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.862861 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-kube-api-access-894x5" pod "02821c8b-2af5-47f6-b552-ea28274991b3" (UID: "02821c8b-2af5-47f6-b552-ea28274991b3"). InnerVolumeSpecName "kube-api-access-894x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:50.963504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.963470 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:50.963504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:50.963498 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894x5\" (UniqueName: \"kubernetes.io/projected/02821c8b-2af5-47f6-b552-ea28274991b3-kube-api-access-894x5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:29:51.486386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.486347 5591 generic.go:356] "Generic (PLEG): container finished" podID="02821c8b-2af5-47f6-b552-ea28274991b3" containerID="bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4" exitCode=0 Nov 02 14:29:51.486885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.486400 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" Nov 02 14:29:51.486885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.486434 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" event={"ID":"02821c8b-2af5-47f6-b552-ea28274991b3","Type":"ContainerDied","Data":"bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4"} Nov 02 14:29:51.486885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.486492 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln" event={"ID":"02821c8b-2af5-47f6-b552-ea28274991b3","Type":"ContainerDied","Data":"9d8284b690557b8c8ff1212481e7fce376b58cb7e60d8c5643abe7ce9fb3479a"} Nov 02 14:29:51.486885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.486517 5591 scope.go:122] "RemoveContainer" containerID="bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4" Nov 02 14:29:51.499801 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.499778 5591 scope.go:122] "RemoveContainer" containerID="bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4" Nov 02 14:29:51.500244 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:29:51.500220 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4\": container with ID starting with bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4 not found: ID does not exist" containerID="bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4" Nov 02 14:29:51.500342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.500252 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4"} err="failed to get container status \"bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4\": rpc error: code = NotFound desc = could not find container \"bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4\": container with ID starting with bfde6fafa3e80bf81ced6f6a7fd1f87c8d866e522ef73d6024b97f94a02571c4 not found: ID does not exist" Nov 02 14:29:51.542168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.542137 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln"] Nov 02 14:29:51.554905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.554876 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486b6sln"] Nov 02 14:29:51.940750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:51.940664 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:53.432327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:53.432291 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02821c8b-2af5-47f6-b552-ea28274991b3" path="/var/lib/kubelet/pods/02821c8b-2af5-47f6-b552-ea28274991b3/volumes" Nov 02 14:29:53.571429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:53.569389 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:54.819585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:54.819551 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:54.987818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:54.987787 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:55.559061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:55.558093 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:56.234509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.234475 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:56.368488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.368431 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44"] Nov 02 14:29:56.439817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.439784 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm"] Nov 02 14:29:56.440509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.440494 5591 cpu_manager.go:414] "containerMap: removing container" podUID="02821c8b-2af5-47f6-b552-ea28274991b3" containerName="payload-processing" Nov 02 14:29:56.440571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.440510 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02821c8b-2af5-47f6-b552-ea28274991b3" containerName="payload-processing" Nov 02 14:29:56.440571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.440563 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" containerName="payload-processing" Nov 02 14:29:56.440571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.440569 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" containerName="payload-processing" Nov 02 14:29:56.440678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.440671 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ffbb32-5e9f-4f12-9d85-4b2d4cade1e8" containerName="payload-processing" Nov 02 14:29:56.440712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.440681 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="02821c8b-2af5-47f6-b552-ea28274991b3" containerName="payload-processing" Nov 02 14:29:56.445564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.445542 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.462260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.462229 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e8e9b2073ad5\"" Nov 02 14:29:56.463418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.463383 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm"] Nov 02 14:29:56.466834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.466810 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 14:29:56.466834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.466820 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 14:29:56.467007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.466831 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 14:29:56.467007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.466874 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 14:29:56.563518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.563426 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-glvff\"" Nov 02 14:29:56.612011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.602174 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcslz\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-kube-api-access-rcslz\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.612011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.602276 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-plugins-config-volume\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.612011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.602350 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-routing-overlay-volume\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.612011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.602381 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-provider-credentials\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.704663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.704620 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-routing-overlay-volume\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.705733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.705709 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-provider-credentials\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.706627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.706603 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcslz\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-kube-api-access-rcslz\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.707171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.707150 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-plugins-config-volume\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.708267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.708070 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-plugins-config-volume\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.708415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.705643 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-routing-overlay-volume\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.725824 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.725789 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-provider-credentials\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.743336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.743266 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcslz\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-kube-api-access-rcslz\") pod \"payload-processing-external-model-8675f87ddd-rsmbm\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:56.771300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:56.771255 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:57.092614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:57.092578 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm"] Nov 02 14:29:57.093173 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:29:57.093147 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9a965a_6c66_438d_a2f3_517472bdc7bc.slice/crio-6d66b5c7603ee0762dd0abfb425c59c6f4e8cd8f48ed8bc107dabde354efe040 WatchSource:0}: Error finding container 6d66b5c7603ee0762dd0abfb425c59c6f4e8cd8f48ed8bc107dabde354efe040: Status 404 returned error can't find the container with id 6d66b5c7603ee0762dd0abfb425c59c6f4e8cd8f48ed8bc107dabde354efe040 Nov 02 14:29:57.366069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:57.364789 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:57.531212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:57.531174 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" event={"ID":"df9a965a-6c66-438d-a2f3-517472bdc7bc","Type":"ContainerStarted","Data":"1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724"} Nov 02 14:29:57.531412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:57.531240 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" event={"ID":"df9a965a-6c66-438d-a2f3-517472bdc7bc","Type":"ContainerStarted","Data":"6d66b5c7603ee0762dd0abfb425c59c6f4e8cd8f48ed8bc107dabde354efe040"} Nov 02 14:29:57.531412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:57.531266 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:29:57.577066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:57.577013 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" podStartSLOduration=1.576994895 podStartE2EDuration="1.576994895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:57.573652193 +0000 UTC m=+2746.299273963" watchObservedRunningTime="2026-11-02 14:29:57.576994895 +0000 UTC m=+2746.302616658" Nov 02 14:29:57.904354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:57.904296 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:58.131466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:58.131428 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:58.924686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:58.924651 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2"] Nov 02 14:29:58.929203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:58.929185 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:58.963368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:58.963333 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2"] Nov 02 14:29:59.054099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.054057 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.054286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.054193 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.054331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.054283 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvdc\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-kube-api-access-qnvdc\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.054370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.054341 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.155968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.155925 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.156213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.156074 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.156213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.156127 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvdc\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-kube-api-access-qnvdc\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.156213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.156187 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.156816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.156779 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.156932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.156840 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.158481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.158460 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.181164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.181099 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvdc\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-kube-api-access-qnvdc\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.244171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.244138 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:29:59.439032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.438843 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:29:59.555431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:29:59.555395 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2"] Nov 02 14:29:59.556173 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:29:59.556148 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a1b14a_38ba_42ee_b35c_3342eb1e13c9.slice/crio-d721cf410fe0146b13c5f29664ea8cb64b80c4fab4659f8203d025288434bea6 WatchSource:0}: Error finding container d721cf410fe0146b13c5f29664ea8cb64b80c4fab4659f8203d025288434bea6: Status 404 returned error can't find the container with id d721cf410fe0146b13c5f29664ea8cb64b80c4fab4659f8203d025288434bea6 Nov 02 14:30:00.258284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.258237 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l"] Nov 02 14:30:00.264232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.264201 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l"] Nov 02 14:30:00.264402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.264335 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:00.269508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.269482 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:00.288568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.288540 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\"" Nov 02 14:30:00.294211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.294182 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l"] Nov 02 14:30:00.307571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.305844 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l"] Nov 02 14:30:00.311731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.311707 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2"] Nov 02 14:30:00.316421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.316397 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:00.332336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.332310 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m"] Nov 02 14:30:00.336822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.336801 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:00.353515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.353485 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2"] Nov 02 14:30:00.374367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.374324 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5b4m\" (UniqueName: \"kubernetes.io/projected/c7a71b7e-370a-4232-a9e5-f332a8fdb279-kube-api-access-c5b4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l\" (UID: \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:00.374537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.374459 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a71b7e-370a-4232-a9e5-f332a8fdb279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l\" (UID: \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:00.374599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.374562 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7770092-321b-44d6-b18e-5f8268ecc9f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893830-c29q2\" (UID: \"c7770092-321b-44d6-b18e-5f8268ecc9f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:00.374599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.374593 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgrg\" (UniqueName: \"kubernetes.io/projected/c7770092-321b-44d6-b18e-5f8268ecc9f9-kube-api-access-zjgrg\") pod \"maas-api-key-cleanup-29893830-c29q2\" (UID: \"c7770092-321b-44d6-b18e-5f8268ecc9f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:00.374693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.374656 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8659da97-3f58-48b2-84f1-a6d0edda1521-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l\" (UID: \"8659da97-3f58-48b2-84f1-a6d0edda1521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:00.374776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.374761 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddjwk\" (UniqueName: \"kubernetes.io/projected/8659da97-3f58-48b2-84f1-a6d0edda1521-kube-api-access-ddjwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l\" (UID: \"8659da97-3f58-48b2-84f1-a6d0edda1521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:00.375960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.375933 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m"] Nov 02 14:30:00.382008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.381773 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd"] Nov 02 14:30:00.388024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.387878 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8"] Nov 02 14:30:00.404852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.388307 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:00.412520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.412491 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv"] Nov 02 14:30:00.412676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.412656 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:00.424047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.421733 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:00.424047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.422814 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp"] Nov 02 14:30:00.428362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.428231 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:00.428562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.428536 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl"] Nov 02 14:30:00.453853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.453817 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd"] Nov 02 14:30:00.454067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.453962 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:00.463404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.463019 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8"] Nov 02 14:30:00.468491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.468459 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv"] Nov 02 14:30:00.476736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.476680 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239e8a6-5379-44a0-9588-385711ac8f51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m\" (UID: \"9239e8a6-5379-44a0-9588-385711ac8f51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:00.477001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.476965 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpv8h\" (UniqueName: \"kubernetes.io/projected/c4f7180c-435b-4256-aa78-c27b8df3beab-kube-api-access-gpv8h\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd\" (UID: \"c4f7180c-435b-4256-aa78-c27b8df3beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:00.477277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477215 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a71b7e-370a-4232-a9e5-f332a8fdb279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l\" (UID: \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:00.477395 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477369 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2719297b-7496-4fdc-a2c4-1d305727819b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8\" (UID: \"2719297b-7496-4fdc-a2c4-1d305727819b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:00.477463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477431 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2pp\" (UniqueName: \"kubernetes.io/projected/9239e8a6-5379-44a0-9588-385711ac8f51-kube-api-access-rk2pp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m\" (UID: \"9239e8a6-5379-44a0-9588-385711ac8f51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:00.477537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477486 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7770092-321b-44d6-b18e-5f8268ecc9f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893830-c29q2\" (UID: \"c7770092-321b-44d6-b18e-5f8268ecc9f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:00.477537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477525 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgrg\" (UniqueName: \"kubernetes.io/projected/c7770092-321b-44d6-b18e-5f8268ecc9f9-kube-api-access-zjgrg\") pod \"maas-api-key-cleanup-29893830-c29q2\" (UID: \"c7770092-321b-44d6-b18e-5f8268ecc9f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:00.477651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477560 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f7180c-435b-4256-aa78-c27b8df3beab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd\" (UID: \"c4f7180c-435b-4256-aa78-c27b8df3beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:00.477651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477636 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8659da97-3f58-48b2-84f1-a6d0edda1521-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l\" (UID: \"8659da97-3f58-48b2-84f1-a6d0edda1521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:00.477765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477714 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dmvw\" (UniqueName: \"kubernetes.io/projected/2719297b-7496-4fdc-a2c4-1d305727819b-kube-api-access-5dmvw\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8\" (UID: \"2719297b-7496-4fdc-a2c4-1d305727819b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:00.477829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.477777 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddjwk\" (UniqueName: \"kubernetes.io/projected/8659da97-3f58-48b2-84f1-a6d0edda1521-kube-api-access-ddjwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l\" (UID: \"8659da97-3f58-48b2-84f1-a6d0edda1521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:00.478086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.478063 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a71b7e-370a-4232-a9e5-f332a8fdb279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l\" (UID: \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:00.478198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.478108 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5b4m\" (UniqueName: \"kubernetes.io/projected/c7a71b7e-370a-4232-a9e5-f332a8fdb279-kube-api-access-c5b4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l\" (UID: \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:00.479107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.479086 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7770092-321b-44d6-b18e-5f8268ecc9f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893830-c29q2\" (UID: \"c7770092-321b-44d6-b18e-5f8268ecc9f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:00.479883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.479860 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8659da97-3f58-48b2-84f1-a6d0edda1521-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l\" (UID: \"8659da97-3f58-48b2-84f1-a6d0edda1521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:00.488122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.488092 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl"] Nov 02 14:30:00.493166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.493132 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp"] Nov 02 14:30:00.498180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.498155 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj"] Nov 02 14:30:00.503490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.503378 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:00.510347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.510287 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj"] Nov 02 14:30:00.526322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.526292 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddjwk\" (UniqueName: \"kubernetes.io/projected/8659da97-3f58-48b2-84f1-a6d0edda1521-kube-api-access-ddjwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l\" (UID: \"8659da97-3f58-48b2-84f1-a6d0edda1521\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:00.526876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.526851 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5b4m\" (UniqueName: \"kubernetes.io/projected/c7a71b7e-370a-4232-a9e5-f332a8fdb279-kube-api-access-c5b4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l\" (UID: \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:00.527631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.527608 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgrg\" (UniqueName: \"kubernetes.io/projected/c7770092-321b-44d6-b18e-5f8268ecc9f9-kube-api-access-zjgrg\") pod \"maas-api-key-cleanup-29893830-c29q2\" (UID: \"c7770092-321b-44d6-b18e-5f8268ecc9f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:00.555969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.555866 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" event={"ID":"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9","Type":"ContainerStarted","Data":"19e1f05ef28a8f5254c5f370d2a96572ff32bc4c9b2a8bb3aaeb29a9f10622e1"} Nov 02 14:30:00.555969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.555935 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" event={"ID":"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9","Type":"ContainerStarted","Data":"d721cf410fe0146b13c5f29664ea8cb64b80c4fab4659f8203d025288434bea6"} Nov 02 14:30:00.556234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.556064 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:30:00.579965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.579932 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjxt\" (UniqueName: \"kubernetes.io/projected/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-kube-api-access-lkjxt\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl\" (UID: \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:00.580109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580048 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239e8a6-5379-44a0-9588-385711ac8f51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m\" (UID: \"9239e8a6-5379-44a0-9588-385711ac8f51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:00.580109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580095 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpv8h\" (UniqueName: \"kubernetes.io/projected/c4f7180c-435b-4256-aa78-c27b8df3beab-kube-api-access-gpv8h\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd\" (UID: \"c4f7180c-435b-4256-aa78-c27b8df3beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:00.580230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580211 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqsn\" (UniqueName: \"kubernetes.io/projected/8e7aeeb8-0306-4a22-8ef4-7376401f776c-kube-api-access-zwqsn\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj\" (UID: \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:00.580266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580246 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aeeb8-0306-4a22-8ef4-7376401f776c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj\" (UID: \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:00.580368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580340 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2719297b-7496-4fdc-a2c4-1d305727819b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8\" (UID: \"2719297b-7496-4fdc-a2c4-1d305727819b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:00.580423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580401 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2pp\" (UniqueName: \"kubernetes.io/projected/9239e8a6-5379-44a0-9588-385711ac8f51-kube-api-access-rk2pp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m\" (UID: \"9239e8a6-5379-44a0-9588-385711ac8f51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:00.580543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580424 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4eeb0c5-2792-4561-b625-b0aaac536ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv\" (UID: \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:00.580543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580462 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f7180c-435b-4256-aa78-c27b8df3beab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd\" (UID: \"c4f7180c-435b-4256-aa78-c27b8df3beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:00.580543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580502 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f844x\" (UniqueName: \"kubernetes.io/projected/f4eeb0c5-2792-4561-b625-b0aaac536ea3-kube-api-access-f844x\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv\" (UID: \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:00.580543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580527 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0154d150-d85a-4314-9df9-dbf526ada4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp\" (UID: \"0154d150-d85a-4314-9df9-dbf526ada4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:00.580781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580621 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwdgz\" (UniqueName: \"kubernetes.io/projected/0154d150-d85a-4314-9df9-dbf526ada4b5-kube-api-access-mwdgz\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp\" (UID: \"0154d150-d85a-4314-9df9-dbf526ada4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:00.580781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580680 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dmvw\" (UniqueName: \"kubernetes.io/projected/2719297b-7496-4fdc-a2c4-1d305727819b-kube-api-access-5dmvw\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8\" (UID: \"2719297b-7496-4fdc-a2c4-1d305727819b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:00.580781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580703 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239e8a6-5379-44a0-9588-385711ac8f51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m\" (UID: \"9239e8a6-5379-44a0-9588-385711ac8f51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:00.580961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.580816 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl\" (UID: \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:00.581102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.581083 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f7180c-435b-4256-aa78-c27b8df3beab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd\" (UID: \"c4f7180c-435b-4256-aa78-c27b8df3beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:00.581178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.581128 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2719297b-7496-4fdc-a2c4-1d305727819b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8\" (UID: \"2719297b-7496-4fdc-a2c4-1d305727819b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:00.587539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.587515 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:00.600832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.600805 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:00.610902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.610823 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2pp\" (UniqueName: \"kubernetes.io/projected/9239e8a6-5379-44a0-9588-385711ac8f51-kube-api-access-rk2pp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m\" (UID: \"9239e8a6-5379-44a0-9588-385711ac8f51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:00.619416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.619291 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpv8h\" (UniqueName: \"kubernetes.io/projected/c4f7180c-435b-4256-aa78-c27b8df3beab-kube-api-access-gpv8h\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd\" (UID: \"c4f7180c-435b-4256-aa78-c27b8df3beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:00.619416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.619310 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dmvw\" (UniqueName: \"kubernetes.io/projected/2719297b-7496-4fdc-a2c4-1d305727819b-kube-api-access-5dmvw\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8\" (UID: \"2719297b-7496-4fdc-a2c4-1d305727819b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:00.633595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.633566 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:00.644498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.644438 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" podStartSLOduration=2.644418431 podStartE2EDuration="2.644418431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:00.640727225 +0000 UTC m=+2749.366348995" watchObservedRunningTime="2026-11-02 14:30:00.644418431 +0000 UTC m=+2749.370040203" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.675507 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.682529 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f844x\" (UniqueName: \"kubernetes.io/projected/f4eeb0c5-2792-4561-b625-b0aaac536ea3-kube-api-access-f844x\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv\" (UID: \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.682570 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0154d150-d85a-4314-9df9-dbf526ada4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp\" (UID: \"0154d150-d85a-4314-9df9-dbf526ada4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.682622 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwdgz\" (UniqueName: \"kubernetes.io/projected/0154d150-d85a-4314-9df9-dbf526ada4b5-kube-api-access-mwdgz\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp\" (UID: \"0154d150-d85a-4314-9df9-dbf526ada4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.682836 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl\" (UID: \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.682881 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjxt\" (UniqueName: \"kubernetes.io/projected/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-kube-api-access-lkjxt\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl\" (UID: \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.683071 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqsn\" (UniqueName: \"kubernetes.io/projected/8e7aeeb8-0306-4a22-8ef4-7376401f776c-kube-api-access-zwqsn\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj\" (UID: \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.683102 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aeeb8-0306-4a22-8ef4-7376401f776c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj\" (UID: \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.683168 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4eeb0c5-2792-4561-b625-b0aaac536ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv\" (UID: \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.685015 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0154d150-d85a-4314-9df9-dbf526ada4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp\" (UID: \"0154d150-d85a-4314-9df9-dbf526ada4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.686211 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aeeb8-0306-4a22-8ef4-7376401f776c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj\" (UID: \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:00.686999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.686454 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl\" (UID: \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:00.688366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.688339 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4eeb0c5-2792-4561-b625-b0aaac536ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv\" (UID: \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:00.715924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.715860 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqsn\" (UniqueName: \"kubernetes.io/projected/8e7aeeb8-0306-4a22-8ef4-7376401f776c-kube-api-access-zwqsn\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj\" (UID: \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:00.720841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.720813 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f844x\" (UniqueName: \"kubernetes.io/projected/f4eeb0c5-2792-4561-b625-b0aaac536ea3-kube-api-access-f844x\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv\" (UID: \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:00.720969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.720948 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjxt\" (UniqueName: \"kubernetes.io/projected/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-kube-api-access-lkjxt\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl\" (UID: \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:00.727701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.726871 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:00.749062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.748449 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwdgz\" (UniqueName: \"kubernetes.io/projected/0154d150-d85a-4314-9df9-dbf526ada4b5-kube-api-access-mwdgz\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp\" (UID: \"0154d150-d85a-4314-9df9-dbf526ada4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:00.795303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.795094 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:00.805019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.804996 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:00.822431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.821230 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:00.864377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.864339 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:00.871289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.870264 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:00.933315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:00.932817 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l"] Nov 02 14:30:00.936622 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:00.936576 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a71b7e_370a_4232_a9e5_f332a8fdb279.slice/crio-0b915eb82a0fe22cd3b8b6db003cacbdca95ed80fd37471d845f6d27e9798281 WatchSource:0}: Error finding container 0b915eb82a0fe22cd3b8b6db003cacbdca95ed80fd37471d845f6d27e9798281: Status 404 returned error can't find the container with id 0b915eb82a0fe22cd3b8b6db003cacbdca95ed80fd37471d845f6d27e9798281 Nov 02 14:30:01.388084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.363712 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l"] Nov 02 14:30:01.388084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.369355 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" podUID="7dbbf5c4-8864-4a37-b772-a62538a5e622" containerName="payload-processing" containerID="cri-o://5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55" gracePeriod=25 Nov 02 14:30:01.407646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.407597 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m"] Nov 02 14:30:01.680995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.666133 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:01.715403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.702498 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2"] Nov 02 14:30:01.785726 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:01.781718 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9239e8a6_5379_44a0_9588_385711ac8f51.slice/crio-59e17a1ac1669c52e4db2b579bd857e0bd747c562aa726d5ca53a467b781c098 WatchSource:0}: Error finding container 59e17a1ac1669c52e4db2b579bd857e0bd747c562aa726d5ca53a467b781c098: Status 404 returned error can't find the container with id 59e17a1ac1669c52e4db2b579bd857e0bd747c562aa726d5ca53a467b781c098 Nov 02 14:30:01.790498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.788823 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" event={"ID":"c7a71b7e-370a-4232-a9e5-f332a8fdb279","Type":"ContainerStarted","Data":"0b915eb82a0fe22cd3b8b6db003cacbdca95ed80fd37471d845f6d27e9798281"} Nov 02 14:30:01.803704 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:01.803670 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:30:01.807905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.807107 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" event={"ID":"8659da97-3f58-48b2-84f1-a6d0edda1521","Type":"ContainerStarted","Data":"552c74f63aebc396266fe8518306c0927b902a43ef7c8cdd456fbb25c002a9fe"} Nov 02 14:30:01.908518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.908479 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd"] Nov 02 14:30:01.915701 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:01.915665 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f7180c_435b_4256_aa78_c27b8df3beab.slice/crio-67718ecd143656ba4e0a284545778016bc04365bdb778029d3e0e61b52e9b0c1 WatchSource:0}: Error finding container 67718ecd143656ba4e0a284545778016bc04365bdb778029d3e0e61b52e9b0c1: Status 404 returned error can't find the container with id 67718ecd143656ba4e0a284545778016bc04365bdb778029d3e0e61b52e9b0c1 Nov 02 14:30:01.962678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:01.962627 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv"] Nov 02 14:30:02.193148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.185479 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8"] Nov 02 14:30:02.592135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.592097 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl"] Nov 02 14:30:02.594310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.594286 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:30:02.608509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.608486 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp"] Nov 02 14:30:02.612936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.612233 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj"] Nov 02 14:30:02.697424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.697386 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-provider-credentials\") pod \"7dbbf5c4-8864-4a37-b772-a62538a5e622\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " Nov 02 14:30:02.697682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.697658 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-kube-api-access-6dtvm\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-kube-api-access-6dtvm\") pod \"7dbbf5c4-8864-4a37-b772-a62538a5e622\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " Nov 02 14:30:02.697833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.697810 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-routing-overlay-volume\") pod \"7dbbf5c4-8864-4a37-b772-a62538a5e622\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " Nov 02 14:30:02.698070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.698044 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-plugins-config-volume\") pod \"7dbbf5c4-8864-4a37-b772-a62538a5e622\" (UID: \"7dbbf5c4-8864-4a37-b772-a62538a5e622\") " Nov 02 14:30:02.698506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.698425 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-routing-overlay-volume" pod "7dbbf5c4-8864-4a37-b772-a62538a5e622" (UID: "7dbbf5c4-8864-4a37-b772-a62538a5e622"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:02.698833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.698798 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-plugins-config-volume" pod "7dbbf5c4-8864-4a37-b772-a62538a5e622" (UID: "7dbbf5c4-8864-4a37-b772-a62538a5e622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:02.698931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.698825 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:02.739172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.739091 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-provider-credentials" pod "7dbbf5c4-8864-4a37-b772-a62538a5e622" (UID: "7dbbf5c4-8864-4a37-b772-a62538a5e622"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:02.739974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.739586 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-kube-api-access-6dtvm" pod "7dbbf5c4-8864-4a37-b772-a62538a5e622" (UID: "7dbbf5c4-8864-4a37-b772-a62538a5e622"). InnerVolumeSpecName "kube-api-access-6dtvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:02.800466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.800425 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dtvm\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-kube-api-access-6dtvm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:02.800466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.800464 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dbbf5c4-8864-4a37-b772-a62538a5e622-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:02.800652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.800477 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dbbf5c4-8864-4a37-b772-a62538a5e622-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:02.815443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.815399 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" event={"ID":"8e7aeeb8-0306-4a22-8ef4-7376401f776c","Type":"ContainerStarted","Data":"0def35f670b38bbb61a6b7b0d3e1e8cf1c4c4e37efb97f68ca5447ee912a7451"} Nov 02 14:30:02.816973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.816937 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" event={"ID":"3c289825-cc5e-4c02-82ee-5e7eccafb3cc","Type":"ContainerStarted","Data":"10f8d7dd18b2a69c26707dc8c97c688a6189fb25dc65744f299062cfaff57934"} Nov 02 14:30:02.818813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.818785 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" event={"ID":"c7770092-321b-44d6-b18e-5f8268ecc9f9","Type":"ContainerStarted","Data":"41a5f1219dc5803ee4f2a043c6276ca8a3a2a64de21747b449024684a5e969f8"} Nov 02 14:30:02.818915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.818841 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" event={"ID":"c7770092-321b-44d6-b18e-5f8268ecc9f9","Type":"ContainerStarted","Data":"5c56be52c84bb08d05421a7457f7f15541cc807805072bc370aacae3a9e2a7af"} Nov 02 14:30:02.820205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.820175 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" event={"ID":"0154d150-d85a-4314-9df9-dbf526ada4b5","Type":"ContainerStarted","Data":"e45c63734deec8f16e6625eed18f455349dc8d8b3f3ec97038c36178575fa9b5"} Nov 02 14:30:02.821319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.821297 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" event={"ID":"f4eeb0c5-2792-4561-b625-b0aaac536ea3","Type":"ContainerStarted","Data":"c4a2a1b15046c25d249e48c946b816d945dc4aa2b9ec5dd1275a78f7640b4ff9"} Nov 02 14:30:02.822835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.822809 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" event={"ID":"9239e8a6-5379-44a0-9588-385711ac8f51","Type":"ContainerStarted","Data":"d8834ce51881d324737e18b4b975b136961c0fe12ff175d3ef1b99a78d989af8"} Nov 02 14:30:02.822933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.822859 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" event={"ID":"9239e8a6-5379-44a0-9588-385711ac8f51","Type":"ContainerStarted","Data":"59e17a1ac1669c52e4db2b579bd857e0bd747c562aa726d5ca53a467b781c098"} Nov 02 14:30:02.824297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.824272 5591 generic.go:356] "Generic (PLEG): container finished" podID="7dbbf5c4-8864-4a37-b772-a62538a5e622" containerID="5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55" exitCode=0 Nov 02 14:30:02.824431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.824318 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" Nov 02 14:30:02.824431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.824352 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" event={"ID":"7dbbf5c4-8864-4a37-b772-a62538a5e622","Type":"ContainerDied","Data":"5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55"} Nov 02 14:30:02.824431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.824392 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44" event={"ID":"7dbbf5c4-8864-4a37-b772-a62538a5e622","Type":"ContainerDied","Data":"8b1ff1e1ec7ef53ac0abb1610eff282fb2fa4111c2a1683043d221383906ee12"} Nov 02 14:30:02.824431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.824417 5591 scope.go:122] "RemoveContainer" containerID="5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55" Nov 02 14:30:02.826256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.826232 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" event={"ID":"c7a71b7e-370a-4232-a9e5-f332a8fdb279","Type":"ContainerStarted","Data":"9700dacb60063aca0bea40d5803d7da34aa2d67b278264ab5780115ab6ced8df"} Nov 02 14:30:02.827824 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.827795 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" event={"ID":"c4f7180c-435b-4256-aa78-c27b8df3beab","Type":"ContainerStarted","Data":"67718ecd143656ba4e0a284545778016bc04365bdb778029d3e0e61b52e9b0c1"} Nov 02 14:30:02.829631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.829594 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" event={"ID":"8659da97-3f58-48b2-84f1-a6d0edda1521","Type":"ContainerStarted","Data":"d7bbaeefa17dfcf5755def46779f7d19cb1799e7a376f3e552b7cafa629a96ba"} Nov 02 14:30:02.830879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.830856 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" event={"ID":"2719297b-7496-4fdc-a2c4-1d305727819b","Type":"ContainerStarted","Data":"201a5e75747710b7ddd01a2fdd2b5e06c7402dce9339e44fc93cb06a8c63798b"} Nov 02 14:30:02.836896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.836872 5591 scope.go:122] "RemoveContainer" containerID="5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55" Nov 02 14:30:02.837257 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:02.837227 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55\": container with ID starting with 5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55 not found: ID does not exist" containerID="5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55" Nov 02 14:30:02.837366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.837265 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55"} err="failed to get container status \"5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55\": rpc error: code = NotFound desc = could not find container \"5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55\": container with ID starting with 5ffa7e088efd3434acb24bba76a2e417679d4faec91a5f65a431783e0e751a55 not found: ID does not exist" Nov 02 14:30:02.861782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.861736 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" podStartSLOduration=2.861723429 podStartE2EDuration="2.861723429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.860498753 +0000 UTC m=+2751.586120524" watchObservedRunningTime="2026-11-02 14:30:02.861723429 +0000 UTC m=+2751.587345195" Nov 02 14:30:02.904348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.904300 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" podStartSLOduration=2.9042865989999997 podStartE2EDuration="2.904286599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.901069136 +0000 UTC m=+2751.626690905" watchObservedRunningTime="2026-11-02 14:30:02.904286599 +0000 UTC m=+2751.629908369" Nov 02 14:30:02.976224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:02.976171 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" podStartSLOduration=2.976158211 podStartE2EDuration="2.976158211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.972641449 +0000 UTC m=+2751.698263218" watchObservedRunningTime="2026-11-02 14:30:02.976158211 +0000 UTC m=+2751.701779981" Nov 02 14:30:03.024287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.024210 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44"] Nov 02 14:30:03.036494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.036466 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5899bvz44"] Nov 02 14:30:03.082388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.082334 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" podStartSLOduration=3.082319907 podStartE2EDuration="3.082319907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:03.079478864 +0000 UTC m=+2751.805100635" watchObservedRunningTime="2026-11-02 14:30:03.082319907 +0000 UTC m=+2751.807941676" Nov 02 14:30:03.266020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.263089 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:03.439031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.438994 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dbbf5c4-8864-4a37-b772-a62538a5e622" path="/var/lib/kubelet/pods/7dbbf5c4-8864-4a37-b772-a62538a5e622/volumes" Nov 02 14:30:03.845146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.845111 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" event={"ID":"0154d150-d85a-4314-9df9-dbf526ada4b5","Type":"ContainerStarted","Data":"dc3cfae76042d8b078812a91445fd57bbc003cbcd4508a5d088a253b7b861d14"} Nov 02 14:30:03.845536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.845422 5591 scope.go:122] "RemoveContainer" containerID="dc3cfae76042d8b078812a91445fd57bbc003cbcd4508a5d088a253b7b861d14" Nov 02 14:30:03.846903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.846878 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" event={"ID":"f4eeb0c5-2792-4561-b625-b0aaac536ea3","Type":"ContainerStarted","Data":"2b6f6f9e87045c0d141a407c5690dafad2b4a7025b14e6b5d5688e1752493b31"} Nov 02 14:30:03.847955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.847935 5591 scope.go:122] "RemoveContainer" containerID="2b6f6f9e87045c0d141a407c5690dafad2b4a7025b14e6b5d5688e1752493b31" Nov 02 14:30:03.849267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.849237 5591 generic.go:356] "Generic (PLEG): container finished" podID="9239e8a6-5379-44a0-9588-385711ac8f51" containerID="d8834ce51881d324737e18b4b975b136961c0fe12ff175d3ef1b99a78d989af8" exitCode=60 Nov 02 14:30:03.849365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.849342 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" event={"ID":"9239e8a6-5379-44a0-9588-385711ac8f51","Type":"ContainerDied","Data":"d8834ce51881d324737e18b4b975b136961c0fe12ff175d3ef1b99a78d989af8"} Nov 02 14:30:03.849857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.849607 5591 scope.go:122] "RemoveContainer" containerID="d8834ce51881d324737e18b4b975b136961c0fe12ff175d3ef1b99a78d989af8" Nov 02 14:30:03.852565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.852540 5591 generic.go:356] "Generic (PLEG): container finished" podID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerID="9700dacb60063aca0bea40d5803d7da34aa2d67b278264ab5780115ab6ced8df" exitCode=60 Nov 02 14:30:03.852682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.852624 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" event={"ID":"c7a71b7e-370a-4232-a9e5-f332a8fdb279","Type":"ContainerDied","Data":"9700dacb60063aca0bea40d5803d7da34aa2d67b278264ab5780115ab6ced8df"} Nov 02 14:30:03.852866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.852851 5591 scope.go:122] "RemoveContainer" containerID="9700dacb60063aca0bea40d5803d7da34aa2d67b278264ab5780115ab6ced8df" Nov 02 14:30:03.854595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.854570 5591 generic.go:356] "Generic (PLEG): container finished" podID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerID="fcadac84c4fb9efc4c9248dec57da005f400386f902d1f50f17cf127b420e14e" exitCode=60 Nov 02 14:30:03.854677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.854665 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" event={"ID":"c4f7180c-435b-4256-aa78-c27b8df3beab","Type":"ContainerDied","Data":"fcadac84c4fb9efc4c9248dec57da005f400386f902d1f50f17cf127b420e14e"} Nov 02 14:30:03.855021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.855001 5591 scope.go:122] "RemoveContainer" containerID="fcadac84c4fb9efc4c9248dec57da005f400386f902d1f50f17cf127b420e14e" Nov 02 14:30:03.856711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.856689 5591 generic.go:356] "Generic (PLEG): container finished" podID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerID="d7bbaeefa17dfcf5755def46779f7d19cb1799e7a376f3e552b7cafa629a96ba" exitCode=60 Nov 02 14:30:03.856821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.856767 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" event={"ID":"8659da97-3f58-48b2-84f1-a6d0edda1521","Type":"ContainerDied","Data":"d7bbaeefa17dfcf5755def46779f7d19cb1799e7a376f3e552b7cafa629a96ba"} Nov 02 14:30:03.857032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.857012 5591 scope.go:122] "RemoveContainer" containerID="d7bbaeefa17dfcf5755def46779f7d19cb1799e7a376f3e552b7cafa629a96ba" Nov 02 14:30:03.860413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.860390 5591 generic.go:356] "Generic (PLEG): container finished" podID="2719297b-7496-4fdc-a2c4-1d305727819b" containerID="de1470156e212cfd1d2ddfe5d43d0d1b23e2934847c8594547c2394919a377e2" exitCode=60 Nov 02 14:30:03.860560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.860499 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" event={"ID":"2719297b-7496-4fdc-a2c4-1d305727819b","Type":"ContainerDied","Data":"de1470156e212cfd1d2ddfe5d43d0d1b23e2934847c8594547c2394919a377e2"} Nov 02 14:30:03.860818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.860799 5591 scope.go:122] "RemoveContainer" containerID="de1470156e212cfd1d2ddfe5d43d0d1b23e2934847c8594547c2394919a377e2" Nov 02 14:30:03.862723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.862705 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" event={"ID":"8e7aeeb8-0306-4a22-8ef4-7376401f776c","Type":"ContainerStarted","Data":"014ee6234cf0325c77eae56911b3b3405b46b6caceed83d17aa01f64843ea928"} Nov 02 14:30:03.863045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.863026 5591 scope.go:122] "RemoveContainer" containerID="014ee6234cf0325c77eae56911b3b3405b46b6caceed83d17aa01f64843ea928" Nov 02 14:30:03.865077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.864532 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" event={"ID":"3c289825-cc5e-4c02-82ee-5e7eccafb3cc","Type":"ContainerStarted","Data":"3d60f93eba5df98ca0963d62ca8d65656bc01dae5488d2a7587e618acee88415"} Nov 02 14:30:03.865077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.864779 5591 scope.go:122] "RemoveContainer" containerID="3d60f93eba5df98ca0963d62ca8d65656bc01dae5488d2a7587e618acee88415" Nov 02 14:30:03.866377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.866355 5591 generic.go:356] "Generic (PLEG): container finished" podID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerID="41a5f1219dc5803ee4f2a043c6276ca8a3a2a64de21747b449024684a5e969f8" exitCode=60 Nov 02 14:30:03.866482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.866382 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" event={"ID":"c7770092-321b-44d6-b18e-5f8268ecc9f9","Type":"ContainerDied","Data":"41a5f1219dc5803ee4f2a043c6276ca8a3a2a64de21747b449024684a5e969f8"} Nov 02 14:30:03.866658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:03.866639 5591 scope.go:122] "RemoveContainer" containerID="41a5f1219dc5803ee4f2a043c6276ca8a3a2a64de21747b449024684a5e969f8" Nov 02 14:30:04.876821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.876780 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" event={"ID":"c7a71b7e-370a-4232-a9e5-f332a8fdb279","Type":"ContainerStarted","Data":"b63227efdbff30193ebb432b7959eee9290ccd60c2429752c22475335b242d42"} Nov 02 14:30:04.880519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.880496 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" event={"ID":"c4f7180c-435b-4256-aa78-c27b8df3beab","Type":"ContainerStarted","Data":"e23bc9bcd0df830dc8829d9efc36d660f5c28169751252525a47c2a5eaef70d2"} Nov 02 14:30:04.883470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.883441 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" event={"ID":"8659da97-3f58-48b2-84f1-a6d0edda1521","Type":"ContainerStarted","Data":"a8db5e73e17af655b3483f84aa1b0e0e655f0b244ce62d968e47a3b32d1f9df3"} Nov 02 14:30:04.885477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.885450 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" event={"ID":"2719297b-7496-4fdc-a2c4-1d305727819b","Type":"ContainerStarted","Data":"9f1d01943fd3349b890cb1dacb1f854669e7c3d2610f2e2e1a1ce85de3b0e1c6"} Nov 02 14:30:04.887344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.887312 5591 generic.go:356] "Generic (PLEG): container finished" podID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerID="014ee6234cf0325c77eae56911b3b3405b46b6caceed83d17aa01f64843ea928" exitCode=60 Nov 02 14:30:04.887448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.887393 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" event={"ID":"8e7aeeb8-0306-4a22-8ef4-7376401f776c","Type":"ContainerDied","Data":"014ee6234cf0325c77eae56911b3b3405b46b6caceed83d17aa01f64843ea928"} Nov 02 14:30:04.887448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.887440 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" event={"ID":"8e7aeeb8-0306-4a22-8ef4-7376401f776c","Type":"ContainerStarted","Data":"bf959e3399fc3430ec908b9e9cbf0baa00a8ca25c6c004fa8b9acdcf09aec8e6"} Nov 02 14:30:04.889319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.889299 5591 generic.go:356] "Generic (PLEG): container finished" podID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerID="3d60f93eba5df98ca0963d62ca8d65656bc01dae5488d2a7587e618acee88415" exitCode=60 Nov 02 14:30:04.889424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.889369 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" event={"ID":"3c289825-cc5e-4c02-82ee-5e7eccafb3cc","Type":"ContainerDied","Data":"3d60f93eba5df98ca0963d62ca8d65656bc01dae5488d2a7587e618acee88415"} Nov 02 14:30:04.889424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.889415 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" event={"ID":"3c289825-cc5e-4c02-82ee-5e7eccafb3cc","Type":"ContainerStarted","Data":"a99280f6b08a616dd970fa0cce0c1e86336e25c88385a31f4808c4993ae36745"} Nov 02 14:30:04.891191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.891168 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" event={"ID":"c7770092-321b-44d6-b18e-5f8268ecc9f9","Type":"ContainerStarted","Data":"3a2f9e8cfcc604366bea5052f7636a42cc932e01f92317c78aa72f6da22721b8"} Nov 02 14:30:04.892808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.892786 5591 generic.go:356] "Generic (PLEG): container finished" podID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerID="dc3cfae76042d8b078812a91445fd57bbc003cbcd4508a5d088a253b7b861d14" exitCode=60 Nov 02 14:30:04.892941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.892847 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" event={"ID":"0154d150-d85a-4314-9df9-dbf526ada4b5","Type":"ContainerDied","Data":"dc3cfae76042d8b078812a91445fd57bbc003cbcd4508a5d088a253b7b861d14"} Nov 02 14:30:04.894564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.894546 5591 generic.go:356] "Generic (PLEG): container finished" podID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerID="2b6f6f9e87045c0d141a407c5690dafad2b4a7025b14e6b5d5688e1752493b31" exitCode=60 Nov 02 14:30:04.894659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.894622 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" event={"ID":"f4eeb0c5-2792-4561-b625-b0aaac536ea3","Type":"ContainerDied","Data":"2b6f6f9e87045c0d141a407c5690dafad2b4a7025b14e6b5d5688e1752493b31"} Nov 02 14:30:04.894724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.894668 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" event={"ID":"f4eeb0c5-2792-4561-b625-b0aaac536ea3","Type":"ContainerStarted","Data":"108e08fb845a7926c4db859dfb44c9fef45a66cf2f233c83f684083e0b7592be"} Nov 02 14:30:04.896366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.896348 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" event={"ID":"9239e8a6-5379-44a0-9588-385711ac8f51","Type":"ContainerStarted","Data":"e9bdc2e3484c904a46a5ab9fb72e4fc6b360f7a866089dffa8197a2cd842bbc4"} Nov 02 14:30:04.997009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.987431 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" podStartSLOduration=4.987410273 podStartE2EDuration="4.987410273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:04.984284425 +0000 UTC m=+2753.709906197" watchObservedRunningTime="2026-11-02 14:30:04.987410273 +0000 UTC m=+2753.713032045" Nov 02 14:30:04.997009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:04.989364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:05.027896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.027808 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" podStartSLOduration=5.027789269 podStartE2EDuration="5.027789269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:05.027001854 +0000 UTC m=+2753.752623622" watchObservedRunningTime="2026-11-02 14:30:05.027789269 +0000 UTC m=+2753.753411042" Nov 02 14:30:05.198224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.198122 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" podStartSLOduration=5.198108732 podStartE2EDuration="5.198108732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:05.19439601 +0000 UTC m=+2753.920017781" watchObservedRunningTime="2026-11-02 14:30:05.198108732 +0000 UTC m=+2753.923730503" Nov 02 14:30:05.236911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.236858 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" podStartSLOduration=5.236838815 podStartE2EDuration="5.236838815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:05.235914708 +0000 UTC m=+2753.961536473" watchObservedRunningTime="2026-11-02 14:30:05.236838815 +0000 UTC m=+2753.962460589" Nov 02 14:30:05.280952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.280900 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" podStartSLOduration=5.280885219 podStartE2EDuration="5.280885219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:05.278368792 +0000 UTC m=+2754.003999260" watchObservedRunningTime="2026-11-02 14:30:05.280885219 +0000 UTC m=+2754.006507013" Nov 02 14:30:05.906863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.906826 5591 generic.go:356] "Generic (PLEG): container finished" podID="9239e8a6-5379-44a0-9588-385711ac8f51" containerID="e9bdc2e3484c904a46a5ab9fb72e4fc6b360f7a866089dffa8197a2cd842bbc4" exitCode=60 Nov 02 14:30:05.907413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.906877 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" event={"ID":"9239e8a6-5379-44a0-9588-385711ac8f51","Type":"ContainerDied","Data":"e9bdc2e3484c904a46a5ab9fb72e4fc6b360f7a866089dffa8197a2cd842bbc4"} Nov 02 14:30:05.907413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.906951 5591 scope.go:122] "RemoveContainer" containerID="d8834ce51881d324737e18b4b975b136961c0fe12ff175d3ef1b99a78d989af8" Nov 02 14:30:05.907413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.907265 5591 scope.go:122] "RemoveContainer" containerID="e9bdc2e3484c904a46a5ab9fb72e4fc6b360f7a866089dffa8197a2cd842bbc4" Nov 02 14:30:05.907724 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.907686 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m_odh-ai-gateway-infra(9239e8a6-5379-44a0-9588-385711ac8f51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" Nov 02 14:30:05.910382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.910361 5591 generic.go:356] "Generic (PLEG): container finished" podID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerID="b63227efdbff30193ebb432b7959eee9290ccd60c2429752c22475335b242d42" exitCode=60 Nov 02 14:30:05.910482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.910431 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" event={"ID":"c7a71b7e-370a-4232-a9e5-f332a8fdb279","Type":"ContainerDied","Data":"b63227efdbff30193ebb432b7959eee9290ccd60c2429752c22475335b242d42"} Nov 02 14:30:05.910751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.910730 5591 scope.go:122] "RemoveContainer" containerID="b63227efdbff30193ebb432b7959eee9290ccd60c2429752c22475335b242d42" Nov 02 14:30:05.911156 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.911124 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l_odh-ai-gateway-infra(c7a71b7e-370a-4232-a9e5-f332a8fdb279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" Nov 02 14:30:05.915256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.915233 5591 generic.go:356] "Generic (PLEG): container finished" podID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerID="e23bc9bcd0df830dc8829d9efc36d660f5c28169751252525a47c2a5eaef70d2" exitCode=60 Nov 02 14:30:05.915366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.915301 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" event={"ID":"c4f7180c-435b-4256-aa78-c27b8df3beab","Type":"ContainerDied","Data":"e23bc9bcd0df830dc8829d9efc36d660f5c28169751252525a47c2a5eaef70d2"} Nov 02 14:30:05.915648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.915629 5591 scope.go:122] "RemoveContainer" containerID="e23bc9bcd0df830dc8829d9efc36d660f5c28169751252525a47c2a5eaef70d2" Nov 02 14:30:05.916077 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.916052 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd_odh-ai-gateway-infra(c4f7180c-435b-4256-aa78-c27b8df3beab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" Nov 02 14:30:05.917412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.917392 5591 generic.go:356] "Generic (PLEG): container finished" podID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerID="a8db5e73e17af655b3483f84aa1b0e0e655f0b244ce62d968e47a3b32d1f9df3" exitCode=60 Nov 02 14:30:05.917519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.917466 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" event={"ID":"8659da97-3f58-48b2-84f1-a6d0edda1521","Type":"ContainerDied","Data":"a8db5e73e17af655b3483f84aa1b0e0e655f0b244ce62d968e47a3b32d1f9df3"} Nov 02 14:30:05.917697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.917681 5591 scope.go:122] "RemoveContainer" containerID="a8db5e73e17af655b3483f84aa1b0e0e655f0b244ce62d968e47a3b32d1f9df3" Nov 02 14:30:05.918022 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.917961 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l_odh-ai-gateway-infra(8659da97-3f58-48b2-84f1-a6d0edda1521)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" Nov 02 14:30:05.919276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.919257 5591 generic.go:356] "Generic (PLEG): container finished" podID="2719297b-7496-4fdc-a2c4-1d305727819b" containerID="9f1d01943fd3349b890cb1dacb1f854669e7c3d2610f2e2e1a1ce85de3b0e1c6" exitCode=60 Nov 02 14:30:05.919355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.919336 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" event={"ID":"2719297b-7496-4fdc-a2c4-1d305727819b","Type":"ContainerDied","Data":"9f1d01943fd3349b890cb1dacb1f854669e7c3d2610f2e2e1a1ce85de3b0e1c6"} Nov 02 14:30:05.919632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.919610 5591 scope.go:122] "RemoveContainer" containerID="9f1d01943fd3349b890cb1dacb1f854669e7c3d2610f2e2e1a1ce85de3b0e1c6" Nov 02 14:30:05.919920 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.919899 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8_odh-ai-gateway-infra(2719297b-7496-4fdc-a2c4-1d305727819b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" Nov 02 14:30:05.921267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.921246 5591 generic.go:356] "Generic (PLEG): container finished" podID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerID="bf959e3399fc3430ec908b9e9cbf0baa00a8ca25c6c004fa8b9acdcf09aec8e6" exitCode=60 Nov 02 14:30:05.921362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.921320 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" event={"ID":"8e7aeeb8-0306-4a22-8ef4-7376401f776c","Type":"ContainerDied","Data":"bf959e3399fc3430ec908b9e9cbf0baa00a8ca25c6c004fa8b9acdcf09aec8e6"} Nov 02 14:30:05.921541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.921528 5591 scope.go:122] "RemoveContainer" containerID="bf959e3399fc3430ec908b9e9cbf0baa00a8ca25c6c004fa8b9acdcf09aec8e6" Nov 02 14:30:05.921813 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.921786 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj_odh-ai-gateway-infra(8e7aeeb8-0306-4a22-8ef4-7376401f776c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" Nov 02 14:30:05.922594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.922576 5591 scope.go:122] "RemoveContainer" containerID="9700dacb60063aca0bea40d5803d7da34aa2d67b278264ab5780115ab6ced8df" Nov 02 14:30:05.923301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.923279 5591 generic.go:356] "Generic (PLEG): container finished" podID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerID="a99280f6b08a616dd970fa0cce0c1e86336e25c88385a31f4808c4993ae36745" exitCode=60 Nov 02 14:30:05.923498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.923473 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" event={"ID":"3c289825-cc5e-4c02-82ee-5e7eccafb3cc","Type":"ContainerDied","Data":"a99280f6b08a616dd970fa0cce0c1e86336e25c88385a31f4808c4993ae36745"} Nov 02 14:30:05.923612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.923558 5591 scope.go:122] "RemoveContainer" containerID="a99280f6b08a616dd970fa0cce0c1e86336e25c88385a31f4808c4993ae36745" Nov 02 14:30:05.923810 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.923792 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl_odh-ai-gateway-infra(3c289825-cc5e-4c02-82ee-5e7eccafb3cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" Nov 02 14:30:05.925233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.925211 5591 generic.go:356] "Generic (PLEG): container finished" podID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerID="3a2f9e8cfcc604366bea5052f7636a42cc932e01f92317c78aa72f6da22721b8" exitCode=60 Nov 02 14:30:05.925306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.925230 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" event={"ID":"c7770092-321b-44d6-b18e-5f8268ecc9f9","Type":"ContainerDied","Data":"3a2f9e8cfcc604366bea5052f7636a42cc932e01f92317c78aa72f6da22721b8"} Nov 02 14:30:05.925517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.925500 5591 scope.go:122] "RemoveContainer" containerID="3a2f9e8cfcc604366bea5052f7636a42cc932e01f92317c78aa72f6da22721b8" Nov 02 14:30:05.925764 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.925744 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893830-c29q2_odh-ai-gateway-infra(c7770092-321b-44d6-b18e-5f8268ecc9f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" Nov 02 14:30:05.927302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.927284 5591 generic.go:356] "Generic (PLEG): container finished" podID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerID="679ea0c121e0c6627e415a3148a179b7b43835f6c34aec5b3f93a065de0ceebb" exitCode=60 Nov 02 14:30:05.927410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.927361 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" event={"ID":"0154d150-d85a-4314-9df9-dbf526ada4b5","Type":"ContainerDied","Data":"679ea0c121e0c6627e415a3148a179b7b43835f6c34aec5b3f93a065de0ceebb"} Nov 02 14:30:05.927614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.927597 5591 scope.go:122] "RemoveContainer" containerID="679ea0c121e0c6627e415a3148a179b7b43835f6c34aec5b3f93a065de0ceebb" Nov 02 14:30:05.927863 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.927841 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp_odh-ai-gateway-infra(0154d150-d85a-4314-9df9-dbf526ada4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" Nov 02 14:30:05.929212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.929192 5591 generic.go:356] "Generic (PLEG): container finished" podID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerID="108e08fb845a7926c4db859dfb44c9fef45a66cf2f233c83f684083e0b7592be" exitCode=60 Nov 02 14:30:05.929324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.929230 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" event={"ID":"f4eeb0c5-2792-4561-b625-b0aaac536ea3","Type":"ContainerDied","Data":"108e08fb845a7926c4db859dfb44c9fef45a66cf2f233c83f684083e0b7592be"} Nov 02 14:30:05.929472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.929455 5591 scope.go:122] "RemoveContainer" containerID="108e08fb845a7926c4db859dfb44c9fef45a66cf2f233c83f684083e0b7592be" Nov 02 14:30:05.929700 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:05.929680 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv_odh-ai-gateway-infra(f4eeb0c5-2792-4561-b625-b0aaac536ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" Nov 02 14:30:05.937012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.936963 5591 scope.go:122] "RemoveContainer" containerID="fcadac84c4fb9efc4c9248dec57da005f400386f902d1f50f17cf127b420e14e" Nov 02 14:30:05.952129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.952041 5591 scope.go:122] "RemoveContainer" containerID="d7bbaeefa17dfcf5755def46779f7d19cb1799e7a376f3e552b7cafa629a96ba" Nov 02 14:30:05.966381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.966362 5591 scope.go:122] "RemoveContainer" containerID="de1470156e212cfd1d2ddfe5d43d0d1b23e2934847c8594547c2394919a377e2" Nov 02 14:30:05.980006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:05.979962 5591 scope.go:122] "RemoveContainer" containerID="014ee6234cf0325c77eae56911b3b3405b46b6caceed83d17aa01f64843ea928" Nov 02 14:30:06.005350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.005325 5591 scope.go:122] "RemoveContainer" containerID="3d60f93eba5df98ca0963d62ca8d65656bc01dae5488d2a7587e618acee88415" Nov 02 14:30:06.026333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.026309 5591 scope.go:122] "RemoveContainer" containerID="41a5f1219dc5803ee4f2a043c6276ca8a3a2a64de21747b449024684a5e969f8" Nov 02 14:30:06.040849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.040829 5591 scope.go:122] "RemoveContainer" containerID="dc3cfae76042d8b078812a91445fd57bbc003cbcd4508a5d088a253b7b861d14" Nov 02 14:30:06.055029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.054973 5591 scope.go:122] "RemoveContainer" containerID="2b6f6f9e87045c0d141a407c5690dafad2b4a7025b14e6b5d5688e1752493b31" Nov 02 14:30:06.940550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.940523 5591 scope.go:122] "RemoveContainer" containerID="bf959e3399fc3430ec908b9e9cbf0baa00a8ca25c6c004fa8b9acdcf09aec8e6" Nov 02 14:30:06.941063 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.940922 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj_odh-ai-gateway-infra(8e7aeeb8-0306-4a22-8ef4-7376401f776c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" Nov 02 14:30:06.942171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.942143 5591 scope.go:122] "RemoveContainer" containerID="a99280f6b08a616dd970fa0cce0c1e86336e25c88385a31f4808c4993ae36745" Nov 02 14:30:06.942525 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.942504 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl_odh-ai-gateway-infra(3c289825-cc5e-4c02-82ee-5e7eccafb3cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" Nov 02 14:30:06.943824 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.943805 5591 scope.go:122] "RemoveContainer" containerID="3a2f9e8cfcc604366bea5052f7636a42cc932e01f92317c78aa72f6da22721b8" Nov 02 14:30:06.944072 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.944052 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893830-c29q2_odh-ai-gateway-infra(c7770092-321b-44d6-b18e-5f8268ecc9f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" Nov 02 14:30:06.945435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.945417 5591 scope.go:122] "RemoveContainer" containerID="679ea0c121e0c6627e415a3148a179b7b43835f6c34aec5b3f93a065de0ceebb" Nov 02 14:30:06.945681 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.945657 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp_odh-ai-gateway-infra(0154d150-d85a-4314-9df9-dbf526ada4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" Nov 02 14:30:06.947184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.947167 5591 scope.go:122] "RemoveContainer" containerID="108e08fb845a7926c4db859dfb44c9fef45a66cf2f233c83f684083e0b7592be" Nov 02 14:30:06.947492 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.947468 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv_odh-ai-gateway-infra(f4eeb0c5-2792-4561-b625-b0aaac536ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" Nov 02 14:30:06.948709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.948689 5591 scope.go:122] "RemoveContainer" containerID="e9bdc2e3484c904a46a5ab9fb72e4fc6b360f7a866089dffa8197a2cd842bbc4" Nov 02 14:30:06.948931 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.948905 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m_odh-ai-gateway-infra(9239e8a6-5379-44a0-9588-385711ac8f51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" Nov 02 14:30:06.950365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.950340 5591 scope.go:122] "RemoveContainer" containerID="b63227efdbff30193ebb432b7959eee9290ccd60c2429752c22475335b242d42" Nov 02 14:30:06.950580 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.950559 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l_odh-ai-gateway-infra(c7a71b7e-370a-4232-a9e5-f332a8fdb279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" Nov 02 14:30:06.951955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.951938 5591 scope.go:122] "RemoveContainer" containerID="e23bc9bcd0df830dc8829d9efc36d660f5c28169751252525a47c2a5eaef70d2" Nov 02 14:30:06.952168 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.952149 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd_odh-ai-gateway-infra(c4f7180c-435b-4256-aa78-c27b8df3beab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" Nov 02 14:30:06.953575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.953556 5591 scope.go:122] "RemoveContainer" containerID="a8db5e73e17af655b3483f84aa1b0e0e655f0b244ce62d968e47a3b32d1f9df3" Nov 02 14:30:06.953827 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.953802 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l_odh-ai-gateway-infra(8659da97-3f58-48b2-84f1-a6d0edda1521)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" Nov 02 14:30:06.955125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:06.955104 5591 scope.go:122] "RemoveContainer" containerID="9f1d01943fd3349b890cb1dacb1f854669e7c3d2610f2e2e1a1ce85de3b0e1c6" Nov 02 14:30:06.955358 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:06.955335 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8_odh-ai-gateway-infra(2719297b-7496-4fdc-a2c4-1d305727819b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" Nov 02 14:30:07.022867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.022829 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:07.099103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.099060 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh"] Nov 02 14:30:07.100038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.100011 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7dbbf5c4-8864-4a37-b772-a62538a5e622" containerName="payload-processing" Nov 02 14:30:07.100038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.100037 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbbf5c4-8864-4a37-b772-a62538a5e622" containerName="payload-processing" Nov 02 14:30:07.100271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.100256 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbbf5c4-8864-4a37-b772-a62538a5e622" containerName="payload-processing" Nov 02 14:30:07.105555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.105534 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.132856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.132826 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh"] Nov 02 14:30:07.206072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.205949 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.206237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.206117 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.206299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.206279 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.206364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.206351 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r2xn\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-kube-api-access-7r2xn\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.206441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.206419 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.206499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.206468 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.206541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.206507 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.206541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.206534 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.206701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.206680 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308242 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r2xn\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-kube-api-access-7r2xn\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308298 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308330 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308364 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308394 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308527 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308588 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308774 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.308972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.308944 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.309110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.309039 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.309175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.309136 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.309381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.309340 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.309494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.309470 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.309600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.309573 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.311023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.310971 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.311507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.311485 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.338036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.337978 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.338253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.338230 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r2xn\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-kube-api-access-7r2xn\") pod \"maas-default-gateway-openshift-default-6886b96f9-77nsh\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.420326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.420240 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:07.773062 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:07.773025 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ea71ac_fe5a_4018_95f7_6a71a4bec2a1.slice/crio-54ba52e9b67c55b4ae54de633b81b975016f4efb7d3ac02e9ddb7db111544a09 WatchSource:0}: Error finding container 54ba52e9b67c55b4ae54de633b81b975016f4efb7d3ac02e9ddb7db111544a09: Status 404 returned error can't find the container with id 54ba52e9b67c55b4ae54de633b81b975016f4efb7d3ac02e9ddb7db111544a09 Nov 02 14:30:07.773567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.773504 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh"] Nov 02 14:30:07.775919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.775888 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:07.776024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.775951 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:07.776114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.776013 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:07.964057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.964021 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" event={"ID":"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1","Type":"ContainerStarted","Data":"597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423"} Nov 02 14:30:07.964592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:07.964091 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" event={"ID":"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1","Type":"ContainerStarted","Data":"54ba52e9b67c55b4ae54de633b81b975016f4efb7d3ac02e9ddb7db111544a09"} Nov 02 14:30:08.012154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:08.012080 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podStartSLOduration=1.012063784 podStartE2EDuration="1.012063784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:08.010420344 +0000 UTC m=+2756.736042128" watchObservedRunningTime="2026-11-02 14:30:08.012063784 +0000 UTC m=+2756.737685554" Nov 02 14:30:08.420484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:08.420440 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:08.422550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:08.422516 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 14:30:08.422699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:08.422569 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 14:30:08.542959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:08.542925 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:30:08.858061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:08.857734 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:09.421128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:09.421097 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 14:30:09.421567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:09.421146 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 14:30:09.961428 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:09.961389 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2"] Nov 02 14:30:09.964904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:09.964877 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:30:10.100410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:10.100374 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:10.266798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:10.266706 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:10.421351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:10.421306 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 14:30:10.421784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:10.421357 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 14:30:10.458712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:10.458674 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:10.922005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:10.916520 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:11.360034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:11.359913 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:11.451500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:11.449112 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 14:30:11.451500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:11.449190 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 14:30:11.574212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:11.574173 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:11.740483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:11.740449 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:12.211010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:12.210636 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:12.420789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:12.420733 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 14:30:12.420789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:12.420783 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 14:30:12.579325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:12.579229 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:12.956008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:12.953122 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:13.420646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:13.420613 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 14:30:13.420819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:13.420670 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 14:30:14.421168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:14.421125 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 14:30:14.421592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:14.421191 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 14:30:14.585638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:14.585583 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:14.961819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:14.961778 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" podUID="f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" containerName="payload-processing" containerID="cri-o://19e1f05ef28a8f5254c5f370d2a96572ff32bc4c9b2a8bb3aaeb29a9f10622e1" gracePeriod=25 Nov 02 14:30:15.105039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.104961 5591 generic.go:356] "Generic (PLEG): container finished" podID="f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" containerID="19e1f05ef28a8f5254c5f370d2a96572ff32bc4c9b2a8bb3aaeb29a9f10622e1" exitCode=0 Nov 02 14:30:15.105203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.105048 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" event={"ID":"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9","Type":"ContainerDied","Data":"19e1f05ef28a8f5254c5f370d2a96572ff32bc4c9b2a8bb3aaeb29a9f10622e1"} Nov 02 14:30:15.263636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.263612 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:30:15.340446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.340404 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-provider-credentials\") pod \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " Nov 02 14:30:15.340653 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.340632 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-plugins-config-volume\") pod \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " Nov 02 14:30:15.340773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.340757 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-kube-api-access-qnvdc\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-kube-api-access-qnvdc\") pod \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " Nov 02 14:30:15.341088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.341047 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-plugins-config-volume" pod "f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" (UID: "f0a1b14a-38ba-42ee-b35c-3342eb1e13c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:15.341206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.341174 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-routing-overlay-volume\") pod \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\" (UID: \"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9\") " Nov 02 14:30:15.341557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.341529 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-routing-overlay-volume" pod "f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" (UID: "f0a1b14a-38ba-42ee-b35c-3342eb1e13c9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:15.342436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.342406 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:15.342546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.342448 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:15.343215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.343188 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-kube-api-access-qnvdc" pod "f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" (UID: "f0a1b14a-38ba-42ee-b35c-3342eb1e13c9"). InnerVolumeSpecName "kube-api-access-qnvdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:15.343320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.343231 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-provider-credentials" pod "f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" (UID: "f0a1b14a-38ba-42ee-b35c-3342eb1e13c9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:15.420647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.420604 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 02 14:30:15.420847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.420754 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 02 14:30:15.444221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.444182 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:15.444221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:15.444219 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvdc\" (UniqueName: \"kubernetes.io/projected/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9-kube-api-access-qnvdc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:16.116880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.116845 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" Nov 02 14:30:16.117070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.116854 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2" event={"ID":"f0a1b14a-38ba-42ee-b35c-3342eb1e13c9","Type":"ContainerDied","Data":"d721cf410fe0146b13c5f29664ea8cb64b80c4fab4659f8203d025288434bea6"} Nov 02 14:30:16.117070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.117032 5591 scope.go:122] "RemoveContainer" containerID="19e1f05ef28a8f5254c5f370d2a96572ff32bc4c9b2a8bb3aaeb29a9f10622e1" Nov 02 14:30:16.173902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.173834 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2"] Nov 02 14:30:16.180947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.180914 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tsmn2"] Nov 02 14:30:16.289836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.289684 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:16.422629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.422595 5591 scope.go:122] "RemoveContainer" containerID="679ea0c121e0c6627e415a3148a179b7b43835f6c34aec5b3f93a065de0ceebb" Nov 02 14:30:16.422861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.422716 5591 scope.go:122] "RemoveContainer" containerID="9f1d01943fd3349b890cb1dacb1f854669e7c3d2610f2e2e1a1ce85de3b0e1c6" Nov 02 14:30:16.422861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.422833 5591 scope.go:122] "RemoveContainer" containerID="3a2f9e8cfcc604366bea5052f7636a42cc932e01f92317c78aa72f6da22721b8" Nov 02 14:30:16.424905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.423318 5591 scope.go:122] "RemoveContainer" containerID="a8db5e73e17af655b3483f84aa1b0e0e655f0b244ce62d968e47a3b32d1f9df3" Nov 02 14:30:16.424905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.423497 5591 scope.go:122] "RemoveContainer" containerID="b63227efdbff30193ebb432b7959eee9290ccd60c2429752c22475335b242d42" Nov 02 14:30:16.424905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.423639 5591 scope.go:122] "RemoveContainer" containerID="a99280f6b08a616dd970fa0cce0c1e86336e25c88385a31f4808c4993ae36745" Nov 02 14:30:16.801222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.801135 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:16.801676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.801624 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:16.803034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:16.803014 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:30:17.126290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:17.126205 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" event={"ID":"0154d150-d85a-4314-9df9-dbf526ada4b5","Type":"ContainerStarted","Data":"25459ea4288728390c486335d535f3fd06cf2aac8058aec711352631fb91e9b6"} Nov 02 14:30:17.129213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:17.129187 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" event={"ID":"c7a71b7e-370a-4232-a9e5-f332a8fdb279","Type":"ContainerStarted","Data":"ddafa1de1a90635c9521300bc07aef48b061169a01b6eaf86046741ac73388ad"} Nov 02 14:30:17.131179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:17.131149 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" event={"ID":"2719297b-7496-4fdc-a2c4-1d305727819b","Type":"ContainerStarted","Data":"e79fe02ff31e255e89d40691a68a43175e5d0adfce98241644200a798367ea34"} Nov 02 14:30:17.177178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:17.177135 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" podStartSLOduration=17.177123013 podStartE2EDuration="17.177123013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:17.176709342 +0000 UTC m=+2765.902331116" watchObservedRunningTime="2026-11-02 14:30:17.177123013 +0000 UTC m=+2765.902744782" Nov 02 14:30:17.422958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:17.422930 5591 scope.go:122] "RemoveContainer" containerID="e23bc9bcd0df830dc8829d9efc36d660f5c28169751252525a47c2a5eaef70d2" Nov 02 14:30:17.423360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:17.423338 5591 scope.go:122] "RemoveContainer" containerID="108e08fb845a7926c4db859dfb44c9fef45a66cf2f233c83f684083e0b7592be" Nov 02 14:30:17.435815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:17.435676 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" path="/var/lib/kubelet/pods/f0a1b14a-38ba-42ee-b35c-3342eb1e13c9/volumes" Nov 02 14:30:18.147968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.147910 5591 generic.go:356] "Generic (PLEG): container finished" podID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerID="ddafa1de1a90635c9521300bc07aef48b061169a01b6eaf86046741ac73388ad" exitCode=60 Nov 02 14:30:18.148437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.147978 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" event={"ID":"c7a71b7e-370a-4232-a9e5-f332a8fdb279","Type":"ContainerDied","Data":"ddafa1de1a90635c9521300bc07aef48b061169a01b6eaf86046741ac73388ad"} Nov 02 14:30:18.148437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.148112 5591 scope.go:122] "RemoveContainer" containerID="b63227efdbff30193ebb432b7959eee9290ccd60c2429752c22475335b242d42" Nov 02 14:30:18.148767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.148738 5591 scope.go:122] "RemoveContainer" containerID="ddafa1de1a90635c9521300bc07aef48b061169a01b6eaf86046741ac73388ad" Nov 02 14:30:18.149506 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:18.149361 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l_odh-ai-gateway-infra(c7a71b7e-370a-4232-a9e5-f332a8fdb279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" Nov 02 14:30:18.150778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.150738 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" event={"ID":"c4f7180c-435b-4256-aa78-c27b8df3beab","Type":"ContainerStarted","Data":"93782895fc499422a867decb01df6a91de00ae3f7f47e2631400caef2fcc950a"} Nov 02 14:30:18.153091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.153067 5591 generic.go:356] "Generic (PLEG): container finished" podID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerID="139dd5e695c72b51df58a7c75a61d9cfba2403edf4a8f931f7bdbc3f2c6ababa" exitCode=60 Nov 02 14:30:18.153198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.153141 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" event={"ID":"8659da97-3f58-48b2-84f1-a6d0edda1521","Type":"ContainerDied","Data":"139dd5e695c72b51df58a7c75a61d9cfba2403edf4a8f931f7bdbc3f2c6ababa"} Nov 02 14:30:18.153574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.153555 5591 scope.go:122] "RemoveContainer" containerID="139dd5e695c72b51df58a7c75a61d9cfba2403edf4a8f931f7bdbc3f2c6ababa" Nov 02 14:30:18.153914 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:18.153888 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l_odh-ai-gateway-infra(8659da97-3f58-48b2-84f1-a6d0edda1521)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" Nov 02 14:30:18.154256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.154235 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:18.155535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.155508 5591 generic.go:356] "Generic (PLEG): container finished" podID="2719297b-7496-4fdc-a2c4-1d305727819b" containerID="e79fe02ff31e255e89d40691a68a43175e5d0adfce98241644200a798367ea34" exitCode=60 Nov 02 14:30:18.155639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.155582 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" event={"ID":"2719297b-7496-4fdc-a2c4-1d305727819b","Type":"ContainerDied","Data":"e79fe02ff31e255e89d40691a68a43175e5d0adfce98241644200a798367ea34"} Nov 02 14:30:18.156122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.156103 5591 scope.go:122] "RemoveContainer" containerID="e79fe02ff31e255e89d40691a68a43175e5d0adfce98241644200a798367ea34" Nov 02 14:30:18.156412 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:18.156387 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8_odh-ai-gateway-infra(2719297b-7496-4fdc-a2c4-1d305727819b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" Nov 02 14:30:18.158202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.158183 5591 generic.go:356] "Generic (PLEG): container finished" podID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerID="f20abf141ad7c5b8940387aaa549645af6e6a83e5ec36ba706e1d2c465f95718" exitCode=60 Nov 02 14:30:18.158279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.158252 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" event={"ID":"3c289825-cc5e-4c02-82ee-5e7eccafb3cc","Type":"ContainerDied","Data":"f20abf141ad7c5b8940387aaa549645af6e6a83e5ec36ba706e1d2c465f95718"} Nov 02 14:30:18.158601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.158585 5591 scope.go:122] "RemoveContainer" containerID="f20abf141ad7c5b8940387aaa549645af6e6a83e5ec36ba706e1d2c465f95718" Nov 02 14:30:18.158821 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:18.158804 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl_odh-ai-gateway-infra(3c289825-cc5e-4c02-82ee-5e7eccafb3cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" Nov 02 14:30:18.160125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.160108 5591 generic.go:356] "Generic (PLEG): container finished" podID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerID="c15a0d5d03915e4506dcf70512ed45a268dca74db695c3a9ee7c0f59bbfeee47" exitCode=60 Nov 02 14:30:18.160210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.160188 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" event={"ID":"c7770092-321b-44d6-b18e-5f8268ecc9f9","Type":"ContainerDied","Data":"c15a0d5d03915e4506dcf70512ed45a268dca74db695c3a9ee7c0f59bbfeee47"} Nov 02 14:30:18.160602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.160582 5591 scope.go:122] "RemoveContainer" containerID="c15a0d5d03915e4506dcf70512ed45a268dca74db695c3a9ee7c0f59bbfeee47" Nov 02 14:30:18.160897 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:18.160872 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893830-c29q2_odh-ai-gateway-infra(c7770092-321b-44d6-b18e-5f8268ecc9f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" Nov 02 14:30:18.162026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.161994 5591 generic.go:356] "Generic (PLEG): container finished" podID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerID="25459ea4288728390c486335d535f3fd06cf2aac8058aec711352631fb91e9b6" exitCode=60 Nov 02 14:30:18.162109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.162032 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" event={"ID":"0154d150-d85a-4314-9df9-dbf526ada4b5","Type":"ContainerDied","Data":"25459ea4288728390c486335d535f3fd06cf2aac8058aec711352631fb91e9b6"} Nov 02 14:30:18.162318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.162304 5591 scope.go:122] "RemoveContainer" containerID="25459ea4288728390c486335d535f3fd06cf2aac8058aec711352631fb91e9b6" Nov 02 14:30:18.162559 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:18.162535 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp_odh-ai-gateway-infra(0154d150-d85a-4314-9df9-dbf526ada4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" Nov 02 14:30:18.164062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.164036 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" event={"ID":"f4eeb0c5-2792-4561-b625-b0aaac536ea3","Type":"ContainerStarted","Data":"3e8fc887eab40b460537da6e56a7817111b8ed6439aa025a1c3b7b3e5e7c967e"} Nov 02 14:30:18.164418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.164398 5591 scope.go:122] "RemoveContainer" containerID="a8db5e73e17af655b3483f84aa1b0e0e655f0b244ce62d968e47a3b32d1f9df3" Nov 02 14:30:18.179174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.179120 5591 scope.go:122] "RemoveContainer" containerID="9f1d01943fd3349b890cb1dacb1f854669e7c3d2610f2e2e1a1ce85de3b0e1c6" Nov 02 14:30:18.191758 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.191734 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp"] Nov 02 14:30:18.193852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.193829 5591 scope.go:122] "RemoveContainer" containerID="a99280f6b08a616dd970fa0cce0c1e86336e25c88385a31f4808c4993ae36745" Nov 02 14:30:18.207632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.207608 5591 scope.go:122] "RemoveContainer" containerID="3a2f9e8cfcc604366bea5052f7636a42cc932e01f92317c78aa72f6da22721b8" Nov 02 14:30:18.221821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.221794 5591 scope.go:122] "RemoveContainer" containerID="679ea0c121e0c6627e415a3148a179b7b43835f6c34aec5b3f93a065de0ceebb" Nov 02 14:30:18.243209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.243162 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8"] Nov 02 14:30:18.286606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.286508 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l"] Nov 02 14:30:18.423295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:18.423266 5591 scope.go:122] "RemoveContainer" containerID="bf959e3399fc3430ec908b9e9cbf0baa00a8ca25c6c004fa8b9acdcf09aec8e6" Nov 02 14:30:19.178085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.178055 5591 generic.go:356] "Generic (PLEG): container finished" podID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerID="3e8fc887eab40b460537da6e56a7817111b8ed6439aa025a1c3b7b3e5e7c967e" exitCode=60 Nov 02 14:30:19.178474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.178145 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" event={"ID":"f4eeb0c5-2792-4561-b625-b0aaac536ea3","Type":"ContainerDied","Data":"3e8fc887eab40b460537da6e56a7817111b8ed6439aa025a1c3b7b3e5e7c967e"} Nov 02 14:30:19.178474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.178235 5591 scope.go:122] "RemoveContainer" containerID="108e08fb845a7926c4db859dfb44c9fef45a66cf2f233c83f684083e0b7592be" Nov 02 14:30:19.178730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.178705 5591 scope.go:122] "RemoveContainer" containerID="3e8fc887eab40b460537da6e56a7817111b8ed6439aa025a1c3b7b3e5e7c967e" Nov 02 14:30:19.179171 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:19.179147 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv_odh-ai-gateway-infra(f4eeb0c5-2792-4561-b625-b0aaac536ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" Nov 02 14:30:19.183327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.183292 5591 generic.go:356] "Generic (PLEG): container finished" podID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerID="93782895fc499422a867decb01df6a91de00ae3f7f47e2631400caef2fcc950a" exitCode=60 Nov 02 14:30:19.183448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.183357 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" event={"ID":"c4f7180c-435b-4256-aa78-c27b8df3beab","Type":"ContainerDied","Data":"93782895fc499422a867decb01df6a91de00ae3f7f47e2631400caef2fcc950a"} Nov 02 14:30:19.183815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.183786 5591 scope.go:122] "RemoveContainer" containerID="93782895fc499422a867decb01df6a91de00ae3f7f47e2631400caef2fcc950a" Nov 02 14:30:19.184201 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:19.184177 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd_odh-ai-gateway-infra(c4f7180c-435b-4256-aa78-c27b8df3beab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" Nov 02 14:30:19.193821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.193757 5591 scope.go:122] "RemoveContainer" containerID="e23bc9bcd0df830dc8829d9efc36d660f5c28169751252525a47c2a5eaef70d2" Nov 02 14:30:19.195547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.195488 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" event={"ID":"8e7aeeb8-0306-4a22-8ef4-7376401f776c","Type":"ContainerStarted","Data":"14db9f6c8bbffed44e1ddf6d230d9aefef64e816177722271cad55d154564981"} Nov 02 14:30:19.254170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.254133 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l"] Nov 02 14:30:19.337500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.337459 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd"] Nov 02 14:30:19.426639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.422372 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:19.426639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.423101 5591 scope.go:122] "RemoveContainer" containerID="e9bdc2e3484c904a46a5ab9fb72e4fc6b360f7a866089dffa8197a2cd842bbc4" Nov 02 14:30:19.483552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.478497 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv"] Nov 02 14:30:19.483552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.479893 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl"] Nov 02 14:30:19.515472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.515441 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2"] Nov 02 14:30:19.548661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.548626 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0154d150-d85a-4314-9df9-dbf526ada4b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0154d150-d85a-4314-9df9-dbf526ada4b5-maas-api-service-ca\") pod \"0154d150-d85a-4314-9df9-dbf526ada4b5\" (UID: \"0154d150-d85a-4314-9df9-dbf526ada4b5\") " Nov 02 14:30:19.548895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.548875 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0154d150-d85a-4314-9df9-dbf526ada4b5-kube-api-access-mwdgz\" (UniqueName: \"kubernetes.io/projected/0154d150-d85a-4314-9df9-dbf526ada4b5-kube-api-access-mwdgz\") pod \"0154d150-d85a-4314-9df9-dbf526ada4b5\" (UID: \"0154d150-d85a-4314-9df9-dbf526ada4b5\") " Nov 02 14:30:19.549211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.549184 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0154d150-d85a-4314-9df9-dbf526ada4b5-maas-api-service-ca" pod "0154d150-d85a-4314-9df9-dbf526ada4b5" (UID: "0154d150-d85a-4314-9df9-dbf526ada4b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:19.549797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.549770 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0154d150-d85a-4314-9df9-dbf526ada4b5-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.551040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.550968 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0154d150-d85a-4314-9df9-dbf526ada4b5-kube-api-access-mwdgz" pod "0154d150-d85a-4314-9df9-dbf526ada4b5" (UID: "0154d150-d85a-4314-9df9-dbf526ada4b5"). InnerVolumeSpecName "kube-api-access-mwdgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:19.601939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.601341 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:19.608775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.608743 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:19.618034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.617979 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:19.654384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.654226 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwdgz\" (UniqueName: \"kubernetes.io/projected/0154d150-d85a-4314-9df9-dbf526ada4b5-kube-api-access-mwdgz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.709432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.709399 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:19.755058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.755022 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a71b7e-370a-4232-a9e5-f332a8fdb279-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a71b7e-370a-4232-a9e5-f332a8fdb279-maas-api-service-ca\") pod \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\" (UID: \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\") " Nov 02 14:30:19.755257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.755076 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2719297b-7496-4fdc-a2c4-1d305727819b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2719297b-7496-4fdc-a2c4-1d305727819b-maas-api-service-ca\") pod \"2719297b-7496-4fdc-a2c4-1d305727819b\" (UID: \"2719297b-7496-4fdc-a2c4-1d305727819b\") " Nov 02 14:30:19.755257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.755223 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8659da97-3f58-48b2-84f1-a6d0edda1521-kube-api-access-ddjwk\" (UniqueName: \"kubernetes.io/projected/8659da97-3f58-48b2-84f1-a6d0edda1521-kube-api-access-ddjwk\") pod \"8659da97-3f58-48b2-84f1-a6d0edda1521\" (UID: \"8659da97-3f58-48b2-84f1-a6d0edda1521\") " Nov 02 14:30:19.755588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.755259 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8659da97-3f58-48b2-84f1-a6d0edda1521-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8659da97-3f58-48b2-84f1-a6d0edda1521-maas-api-service-ca\") pod \"8659da97-3f58-48b2-84f1-a6d0edda1521\" (UID: \"8659da97-3f58-48b2-84f1-a6d0edda1521\") " Nov 02 14:30:19.755710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.755692 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2719297b-7496-4fdc-a2c4-1d305727819b-kube-api-access-5dmvw\" (UniqueName: \"kubernetes.io/projected/2719297b-7496-4fdc-a2c4-1d305727819b-kube-api-access-5dmvw\") pod \"2719297b-7496-4fdc-a2c4-1d305727819b\" (UID: \"2719297b-7496-4fdc-a2c4-1d305727819b\") " Nov 02 14:30:19.755936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.755916 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a71b7e-370a-4232-a9e5-f332a8fdb279-kube-api-access-c5b4m\" (UniqueName: \"kubernetes.io/projected/c7a71b7e-370a-4232-a9e5-f332a8fdb279-kube-api-access-c5b4m\") pod \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\" (UID: \"c7a71b7e-370a-4232-a9e5-f332a8fdb279\") " Nov 02 14:30:19.756046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.755926 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2719297b-7496-4fdc-a2c4-1d305727819b-maas-api-service-ca" pod "2719297b-7496-4fdc-a2c4-1d305727819b" (UID: "2719297b-7496-4fdc-a2c4-1d305727819b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:19.756664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.756636 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2719297b-7496-4fdc-a2c4-1d305727819b-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.760791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.756907 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8659da97-3f58-48b2-84f1-a6d0edda1521-maas-api-service-ca" pod "8659da97-3f58-48b2-84f1-a6d0edda1521" (UID: "8659da97-3f58-48b2-84f1-a6d0edda1521"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:19.760791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.757103 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a71b7e-370a-4232-a9e5-f332a8fdb279-maas-api-service-ca" pod "c7a71b7e-370a-4232-a9e5-f332a8fdb279" (UID: "c7a71b7e-370a-4232-a9e5-f332a8fdb279"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:19.762885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.762855 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8659da97-3f58-48b2-84f1-a6d0edda1521-kube-api-access-ddjwk" pod "8659da97-3f58-48b2-84f1-a6d0edda1521" (UID: "8659da97-3f58-48b2-84f1-a6d0edda1521"). InnerVolumeSpecName "kube-api-access-ddjwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:19.764405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.764307 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2719297b-7496-4fdc-a2c4-1d305727819b-kube-api-access-5dmvw" pod "2719297b-7496-4fdc-a2c4-1d305727819b" (UID: "2719297b-7496-4fdc-a2c4-1d305727819b"). InnerVolumeSpecName "kube-api-access-5dmvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:19.765042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.765015 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a71b7e-370a-4232-a9e5-f332a8fdb279-kube-api-access-c5b4m" pod "c7a71b7e-370a-4232-a9e5-f332a8fdb279" (UID: "c7a71b7e-370a-4232-a9e5-f332a8fdb279"). InnerVolumeSpecName "kube-api-access-c5b4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:19.811729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.811690 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.857734 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-kube-api-access-lkjxt\" (UniqueName: \"kubernetes.io/projected/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-kube-api-access-lkjxt\") pod \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\" (UID: \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\") " Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.858037 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-maas-api-service-ca\") pod \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\" (UID: \"3c289825-cc5e-4c02-82ee-5e7eccafb3cc\") " Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.858965 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5b4m\" (UniqueName: \"kubernetes.io/projected/c7a71b7e-370a-4232-a9e5-f332a8fdb279-kube-api-access-c5b4m\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.859008 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a71b7e-370a-4232-a9e5-f332a8fdb279-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.859023 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddjwk\" (UniqueName: \"kubernetes.io/projected/8659da97-3f58-48b2-84f1-a6d0edda1521-kube-api-access-ddjwk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.859036 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8659da97-3f58-48b2-84f1-a6d0edda1521-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.859049 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dmvw\" (UniqueName: \"kubernetes.io/projected/2719297b-7496-4fdc-a2c4-1d305727819b-kube-api-access-5dmvw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.869190 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-maas-api-service-ca" pod "3c289825-cc5e-4c02-82ee-5e7eccafb3cc" (UID: "3c289825-cc5e-4c02-82ee-5e7eccafb3cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:19.885073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.875727 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:19.890161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.888724 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-kube-api-access-lkjxt" pod "3c289825-cc5e-4c02-82ee-5e7eccafb3cc" (UID: "3c289825-cc5e-4c02-82ee-5e7eccafb3cc"). InnerVolumeSpecName "kube-api-access-lkjxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:19.960086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.959969 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7770092-321b-44d6-b18e-5f8268ecc9f9-kube-api-access-zjgrg\" (UniqueName: \"kubernetes.io/projected/c7770092-321b-44d6-b18e-5f8268ecc9f9-kube-api-access-zjgrg\") pod \"c7770092-321b-44d6-b18e-5f8268ecc9f9\" (UID: \"c7770092-321b-44d6-b18e-5f8268ecc9f9\") " Nov 02 14:30:19.960249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.960084 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7770092-321b-44d6-b18e-5f8268ecc9f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7770092-321b-44d6-b18e-5f8268ecc9f9-maas-api-service-ca\") pod \"c7770092-321b-44d6-b18e-5f8268ecc9f9\" (UID: \"c7770092-321b-44d6-b18e-5f8268ecc9f9\") " Nov 02 14:30:19.960550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.960523 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7770092-321b-44d6-b18e-5f8268ecc9f9-maas-api-service-ca" pod "c7770092-321b-44d6-b18e-5f8268ecc9f9" (UID: "c7770092-321b-44d6-b18e-5f8268ecc9f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:19.960914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.960887 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkjxt\" (UniqueName: \"kubernetes.io/projected/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-kube-api-access-lkjxt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.961013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.960920 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7770092-321b-44d6-b18e-5f8268ecc9f9-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.961013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.960936 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c289825-cc5e-4c02-82ee-5e7eccafb3cc-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.962246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:19.962217 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7770092-321b-44d6-b18e-5f8268ecc9f9-kube-api-access-zjgrg" pod "c7770092-321b-44d6-b18e-5f8268ecc9f9" (UID: "c7770092-321b-44d6-b18e-5f8268ecc9f9"). InnerVolumeSpecName "kube-api-access-zjgrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:20.067645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.067608 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjgrg\" (UniqueName: \"kubernetes.io/projected/c7770092-321b-44d6-b18e-5f8268ecc9f9-kube-api-access-zjgrg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.068273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.068236 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:20.206576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.206537 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" Nov 02 14:30:20.207118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.206542 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l" event={"ID":"c7a71b7e-370a-4232-a9e5-f332a8fdb279","Type":"ContainerDied","Data":"0b915eb82a0fe22cd3b8b6db003cacbdca95ed80fd37471d845f6d27e9798281"} Nov 02 14:30:20.207118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.206761 5591 scope.go:122] "RemoveContainer" containerID="ddafa1de1a90635c9521300bc07aef48b061169a01b6eaf86046741ac73388ad" Nov 02 14:30:20.210617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.210541 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" event={"ID":"8659da97-3f58-48b2-84f1-a6d0edda1521","Type":"ContainerDied","Data":"552c74f63aebc396266fe8518306c0927b902a43ef7c8cdd456fbb25c002a9fe"} Nov 02 14:30:20.210617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.210565 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l" Nov 02 14:30:20.212270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.212240 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" event={"ID":"2719297b-7496-4fdc-a2c4-1d305727819b","Type":"ContainerDied","Data":"201a5e75747710b7ddd01a2fdd2b5e06c7402dce9339e44fc93cb06a8c63798b"} Nov 02 14:30:20.212386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.212351 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8" Nov 02 14:30:20.221289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.221267 5591 scope.go:122] "RemoveContainer" containerID="139dd5e695c72b51df58a7c75a61d9cfba2403edf4a8f931f7bdbc3f2c6ababa" Nov 02 14:30:20.222609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.222583 5591 generic.go:356] "Generic (PLEG): container finished" podID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerID="14db9f6c8bbffed44e1ddf6d230d9aefef64e816177722271cad55d154564981" exitCode=60 Nov 02 14:30:20.222686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.222651 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" event={"ID":"8e7aeeb8-0306-4a22-8ef4-7376401f776c","Type":"ContainerDied","Data":"14db9f6c8bbffed44e1ddf6d230d9aefef64e816177722271cad55d154564981"} Nov 02 14:30:20.223471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.223257 5591 scope.go:122] "RemoveContainer" containerID="14db9f6c8bbffed44e1ddf6d230d9aefef64e816177722271cad55d154564981" Nov 02 14:30:20.223698 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:20.223650 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj_odh-ai-gateway-infra(8e7aeeb8-0306-4a22-8ef4-7376401f776c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" Nov 02 14:30:20.224900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.224880 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" event={"ID":"3c289825-cc5e-4c02-82ee-5e7eccafb3cc","Type":"ContainerDied","Data":"10f8d7dd18b2a69c26707dc8c97c688a6189fb25dc65744f299062cfaff57934"} Nov 02 14:30:20.225051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.224884 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl" Nov 02 14:30:20.226914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.226887 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" event={"ID":"c7770092-321b-44d6-b18e-5f8268ecc9f9","Type":"ContainerDied","Data":"5c56be52c84bb08d05421a7457f7f15541cc807805072bc370aacae3a9e2a7af"} Nov 02 14:30:20.226914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.226941 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2" Nov 02 14:30:20.229612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.229593 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" Nov 02 14:30:20.229752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.229614 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp" event={"ID":"0154d150-d85a-4314-9df9-dbf526ada4b5","Type":"ContainerDied","Data":"e45c63734deec8f16e6625eed18f455349dc8d8b3f3ec97038c36178575fa9b5"} Nov 02 14:30:20.238834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.238754 5591 scope.go:122] "RemoveContainer" containerID="e79fe02ff31e255e89d40691a68a43175e5d0adfce98241644200a798367ea34" Nov 02 14:30:20.240124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.240100 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" event={"ID":"9239e8a6-5379-44a0-9588-385711ac8f51","Type":"ContainerStarted","Data":"b9cb95b7beca1777dd162cd1f3360825b78d3b874bbda32511ddab09ac12acab"} Nov 02 14:30:20.255178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.254374 5591 scope.go:122] "RemoveContainer" containerID="bf959e3399fc3430ec908b9e9cbf0baa00a8ca25c6c004fa8b9acdcf09aec8e6" Nov 02 14:30:20.278903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.278599 5591 scope.go:122] "RemoveContainer" containerID="f20abf141ad7c5b8940387aaa549645af6e6a83e5ec36ba706e1d2c465f95718" Nov 02 14:30:20.281335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.281224 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:20.289593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.289567 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj"] Nov 02 14:30:20.297917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.295553 5591 scope.go:122] "RemoveContainer" containerID="c15a0d5d03915e4506dcf70512ed45a268dca74db695c3a9ee7c0f59bbfeee47" Nov 02 14:30:20.308349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.308320 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l"] Nov 02 14:30:20.320517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.320487 5591 scope.go:122] "RemoveContainer" containerID="25459ea4288728390c486335d535f3fd06cf2aac8058aec711352631fb91e9b6" Nov 02 14:30:20.323067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.323048 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893830-9vj9l"] Nov 02 14:30:20.390195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.389947 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:20.463084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.463022 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:20.495779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.495750 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l"] Nov 02 14:30:20.499321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.499297 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893830-qht4l"] Nov 02 14:30:20.576340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.576304 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4eeb0c5-2792-4561-b625-b0aaac536ea3-kube-api-access-f844x\" (UniqueName: \"kubernetes.io/projected/f4eeb0c5-2792-4561-b625-b0aaac536ea3-kube-api-access-f844x\") pod \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\" (UID: \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\") " Nov 02 14:30:20.576538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.576445 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f7180c-435b-4256-aa78-c27b8df3beab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f7180c-435b-4256-aa78-c27b8df3beab-maas-api-service-ca\") pod \"c4f7180c-435b-4256-aa78-c27b8df3beab\" (UID: \"c4f7180c-435b-4256-aa78-c27b8df3beab\") " Nov 02 14:30:20.576695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.576674 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f7180c-435b-4256-aa78-c27b8df3beab-kube-api-access-gpv8h\" (UniqueName: \"kubernetes.io/projected/c4f7180c-435b-4256-aa78-c27b8df3beab-kube-api-access-gpv8h\") pod \"c4f7180c-435b-4256-aa78-c27b8df3beab\" (UID: \"c4f7180c-435b-4256-aa78-c27b8df3beab\") " Nov 02 14:30:20.576861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.576836 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4eeb0c5-2792-4561-b625-b0aaac536ea3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4eeb0c5-2792-4561-b625-b0aaac536ea3-maas-api-service-ca\") pod \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\" (UID: \"f4eeb0c5-2792-4561-b625-b0aaac536ea3\") " Nov 02 14:30:20.577284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.577252 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4eeb0c5-2792-4561-b625-b0aaac536ea3-maas-api-service-ca" pod "f4eeb0c5-2792-4561-b625-b0aaac536ea3" (UID: "f4eeb0c5-2792-4561-b625-b0aaac536ea3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:20.578049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.578027 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4eeb0c5-2792-4561-b625-b0aaac536ea3-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.578371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.578343 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f7180c-435b-4256-aa78-c27b8df3beab-maas-api-service-ca" pod "c4f7180c-435b-4256-aa78-c27b8df3beab" (UID: "c4f7180c-435b-4256-aa78-c27b8df3beab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:20.579623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.579603 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f7180c-435b-4256-aa78-c27b8df3beab-kube-api-access-gpv8h" pod "c4f7180c-435b-4256-aa78-c27b8df3beab" (UID: "c4f7180c-435b-4256-aa78-c27b8df3beab"). InnerVolumeSpecName "kube-api-access-gpv8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:20.580093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.580067 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4eeb0c5-2792-4561-b625-b0aaac536ea3-kube-api-access-f844x" pod "f4eeb0c5-2792-4561-b625-b0aaac536ea3" (UID: "f4eeb0c5-2792-4561-b625-b0aaac536ea3"). InnerVolumeSpecName "kube-api-access-f844x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:20.609783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.609745 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8"] Nov 02 14:30:20.609977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.609796 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893830-46wn8"] Nov 02 14:30:20.679854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.679821 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f7180c-435b-4256-aa78-c27b8df3beab-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.679854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.679852 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpv8h\" (UniqueName: \"kubernetes.io/projected/c4f7180c-435b-4256-aa78-c27b8df3beab-kube-api-access-gpv8h\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.680114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.679863 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f844x\" (UniqueName: \"kubernetes.io/projected/f4eeb0c5-2792-4561-b625-b0aaac536ea3-kube-api-access-f844x\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.688464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.688432 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl"] Nov 02 14:30:20.692781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.692753 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893830-z56kl"] Nov 02 14:30:20.730328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.730260 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2"] Nov 02 14:30:20.735675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.735647 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-c29q2"] Nov 02 14:30:20.812690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.812652 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp"] Nov 02 14:30:20.817150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:20.817124 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893830-wfknp"] Nov 02 14:30:21.248863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.248824 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" Nov 02 14:30:21.249379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.248840 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd" event={"ID":"c4f7180c-435b-4256-aa78-c27b8df3beab","Type":"ContainerDied","Data":"67718ecd143656ba4e0a284545778016bc04365bdb778029d3e0e61b52e9b0c1"} Nov 02 14:30:21.249379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.249016 5591 scope.go:122] "RemoveContainer" containerID="93782895fc499422a867decb01df6a91de00ae3f7f47e2631400caef2fcc950a" Nov 02 14:30:21.258128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.257762 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" Nov 02 14:30:21.258244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.257747 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv" event={"ID":"f4eeb0c5-2792-4561-b625-b0aaac536ea3","Type":"ContainerDied","Data":"c4a2a1b15046c25d249e48c946b816d945dc4aa2b9ec5dd1275a78f7640b4ff9"} Nov 02 14:30:21.260102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.260079 5591 generic.go:356] "Generic (PLEG): container finished" podID="9239e8a6-5379-44a0-9588-385711ac8f51" containerID="b9cb95b7beca1777dd162cd1f3360825b78d3b874bbda32511ddab09ac12acab" exitCode=60 Nov 02 14:30:21.260205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.260157 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" event={"ID":"9239e8a6-5379-44a0-9588-385711ac8f51","Type":"ContainerDied","Data":"b9cb95b7beca1777dd162cd1f3360825b78d3b874bbda32511ddab09ac12acab"} Nov 02 14:30:21.260648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.260625 5591 scope.go:122] "RemoveContainer" containerID="b9cb95b7beca1777dd162cd1f3360825b78d3b874bbda32511ddab09ac12acab" Nov 02 14:30:21.261039 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:21.260972 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m_odh-ai-gateway-infra(9239e8a6-5379-44a0-9588-385711ac8f51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" Nov 02 14:30:21.262233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.262212 5591 scope.go:122] "RemoveContainer" containerID="3e8fc887eab40b460537da6e56a7817111b8ed6439aa025a1c3b7b3e5e7c967e" Nov 02 14:30:21.272921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.272903 5591 scope.go:122] "RemoveContainer" containerID="e9bdc2e3484c904a46a5ab9fb72e4fc6b360f7a866089dffa8197a2cd842bbc4" Nov 02 14:30:21.434137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.434107 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:21.435911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.435884 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" path="/var/lib/kubelet/pods/0154d150-d85a-4314-9df9-dbf526ada4b5/volumes" Nov 02 14:30:21.436418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.436403 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" path="/var/lib/kubelet/pods/2719297b-7496-4fdc-a2c4-1d305727819b/volumes" Nov 02 14:30:21.436800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.436787 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" path="/var/lib/kubelet/pods/3c289825-cc5e-4c02-82ee-5e7eccafb3cc/volumes" Nov 02 14:30:21.437602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.437583 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" path="/var/lib/kubelet/pods/8659da97-3f58-48b2-84f1-a6d0edda1521/volumes" Nov 02 14:30:21.438208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.438191 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" path="/var/lib/kubelet/pods/c7770092-321b-44d6-b18e-5f8268ecc9f9/volumes" Nov 02 14:30:21.438731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.438716 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" path="/var/lib/kubelet/pods/c7a71b7e-370a-4232-a9e5-f332a8fdb279/volumes" Nov 02 14:30:21.467853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.467815 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd"] Nov 02 14:30:21.473432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.473409 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893830-smhsd"] Nov 02 14:30:21.558650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.558576 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv"] Nov 02 14:30:21.567996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.567955 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893830-86vcv"] Nov 02 14:30:21.593950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.593926 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7aeeb8-0306-4a22-8ef4-7376401f776c-kube-api-access-zwqsn\" (UniqueName: \"kubernetes.io/projected/8e7aeeb8-0306-4a22-8ef4-7376401f776c-kube-api-access-zwqsn\") pod \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\" (UID: \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\") " Nov 02 14:30:21.594123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.593961 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7aeeb8-0306-4a22-8ef4-7376401f776c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aeeb8-0306-4a22-8ef4-7376401f776c-maas-api-service-ca\") pod \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\" (UID: \"8e7aeeb8-0306-4a22-8ef4-7376401f776c\") " Nov 02 14:30:21.594369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.594348 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7aeeb8-0306-4a22-8ef4-7376401f776c-maas-api-service-ca" pod "8e7aeeb8-0306-4a22-8ef4-7376401f776c" (UID: "8e7aeeb8-0306-4a22-8ef4-7376401f776c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:21.594553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.594537 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7aeeb8-0306-4a22-8ef4-7376401f776c-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.596572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.596525 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7aeeb8-0306-4a22-8ef4-7376401f776c-kube-api-access-zwqsn" pod "8e7aeeb8-0306-4a22-8ef4-7376401f776c" (UID: "8e7aeeb8-0306-4a22-8ef4-7376401f776c"). InnerVolumeSpecName "kube-api-access-zwqsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:21.671052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.663438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:21.696285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.696243 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqsn\" (UniqueName: \"kubernetes.io/projected/8e7aeeb8-0306-4a22-8ef4-7376401f776c-kube-api-access-zwqsn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.785086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:21.785051 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m"] Nov 02 14:30:22.274323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.274294 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" Nov 02 14:30:22.274805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.274329 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj" event={"ID":"8e7aeeb8-0306-4a22-8ef4-7376401f776c","Type":"ContainerDied","Data":"0def35f670b38bbb61a6b7b0d3e1e8cf1c4c4e37efb97f68ca5447ee912a7451"} Nov 02 14:30:22.274805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.274393 5591 scope.go:122] "RemoveContainer" containerID="14db9f6c8bbffed44e1ddf6d230d9aefef64e816177722271cad55d154564981" Nov 02 14:30:22.379314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.379272 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj"] Nov 02 14:30:22.388453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.388417 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893830-wrncj"] Nov 02 14:30:22.499810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.499782 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:22.535826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.535747 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9239e8a6-5379-44a0-9588-385711ac8f51-kube-api-access-rk2pp\" (UniqueName: \"kubernetes.io/projected/9239e8a6-5379-44a0-9588-385711ac8f51-kube-api-access-rk2pp\") pod \"9239e8a6-5379-44a0-9588-385711ac8f51\" (UID: \"9239e8a6-5379-44a0-9588-385711ac8f51\") " Nov 02 14:30:22.536044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.535954 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9239e8a6-5379-44a0-9588-385711ac8f51-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239e8a6-5379-44a0-9588-385711ac8f51-maas-api-service-ca\") pod \"9239e8a6-5379-44a0-9588-385711ac8f51\" (UID: \"9239e8a6-5379-44a0-9588-385711ac8f51\") " Nov 02 14:30:22.536322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.536297 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9239e8a6-5379-44a0-9588-385711ac8f51-maas-api-service-ca" pod "9239e8a6-5379-44a0-9588-385711ac8f51" (UID: "9239e8a6-5379-44a0-9588-385711ac8f51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:22.536592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.536573 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239e8a6-5379-44a0-9588-385711ac8f51-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:22.538027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.538003 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9239e8a6-5379-44a0-9588-385711ac8f51-kube-api-access-rk2pp" pod "9239e8a6-5379-44a0-9588-385711ac8f51" (UID: "9239e8a6-5379-44a0-9588-385711ac8f51"). InnerVolumeSpecName "kube-api-access-rk2pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:22.638325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.638288 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk2pp\" (UniqueName: \"kubernetes.io/projected/9239e8a6-5379-44a0-9588-385711ac8f51-kube-api-access-rk2pp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:22.898246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:22.898211 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:23.068834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.068794 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:23.290610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.290516 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" event={"ID":"9239e8a6-5379-44a0-9588-385711ac8f51","Type":"ContainerDied","Data":"59e17a1ac1669c52e4db2b579bd857e0bd747c562aa726d5ca53a467b781c098"} Nov 02 14:30:23.290610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.290584 5591 scope.go:122] "RemoveContainer" containerID="b9cb95b7beca1777dd162cd1f3360825b78d3b874bbda32511ddab09ac12acab" Nov 02 14:30:23.291199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.290698 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m" Nov 02 14:30:23.355156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.355121 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m"] Nov 02 14:30:23.359187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.359158 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893830-75d9m"] Nov 02 14:30:23.650801 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.650757 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" path="/var/lib/kubelet/pods/8e7aeeb8-0306-4a22-8ef4-7376401f776c/volumes" Nov 02 14:30:23.651541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.651509 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" path="/var/lib/kubelet/pods/9239e8a6-5379-44a0-9588-385711ac8f51/volumes" Nov 02 14:30:23.652158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.652132 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" path="/var/lib/kubelet/pods/c4f7180c-435b-4256-aa78-c27b8df3beab/volumes" Nov 02 14:30:23.664308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.664271 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" path="/var/lib/kubelet/pods/f4eeb0c5-2792-4561-b625-b0aaac536ea3/volumes" Nov 02 14:30:23.784613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:23.784506 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:25.393906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:25.393874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:25.672009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:25.671858 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:27.385532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:27.385499 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:30:27.468803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:27.468420 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:27.898717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:27.898687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:28.160231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:28.160139 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:28.897644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:28.897609 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:29.106015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:29.104819 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:29.873830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:29.873796 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:30.497139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:30.497100 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:30.781574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:30.781479 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:31.098219 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:31.098115 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:30:31.163817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:31.163756 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9"] Nov 02 14:30:31.291727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:31.291687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:32.008553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:32.008512 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:32.206008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:32.199679 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:33.360416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.360382 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2"] Nov 02 14:30:33.361114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361099 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:33.361173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361115 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:33.361173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361126 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.361173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361131 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.361173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361144 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:33.361173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361150 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:33.361173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361167 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:33.361173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361172 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361179 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361183 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361191 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361196 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361204 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361209 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361220 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361224 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361233 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" containerName="payload-processing" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361238 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" containerName="payload-processing" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361246 5591 cpu_manager.go:414] "containerMap: removing container" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361250 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361258 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361262 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361269 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361274 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361282 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361286 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361294 5591 cpu_manager.go:414] "containerMap: removing container" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361299 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361306 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361311 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361320 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361324 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361333 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361338 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361349 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361353 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361362 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361366 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361372 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361376 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361390 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361395 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361403 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:33.361398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361407 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361416 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361420 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361428 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361432 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361440 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361444 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361534 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361542 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361550 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361556 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361563 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361568 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361573 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361578 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361583 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361589 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361595 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361601 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361606 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361611 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361617 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361623 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361630 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361635 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361641 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361647 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a1b14a-38ba-42ee-b35c-3342eb1e13c9" containerName="payload-processing" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361653 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2719297b-7496-4fdc-a2c4-1d305727819b" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361659 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361664 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361670 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361676 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:33.362498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.361682 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:33.372008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.368803 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.386709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.386680 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2"] Nov 02 14:30:33.394627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.394599 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-865bae346716\"" Nov 02 14:30:33.394789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.394619 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f8ae8e27fe78\"" Nov 02 14:30:33.474185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.474151 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r777p\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-kube-api-access-r777p\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.474366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.474272 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-provider-credentials\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.474426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.474410 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-plugins-config-volume\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.474524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.474511 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-routing-overlay-volume\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.575962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.575930 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-routing-overlay-volume\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.576161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.576072 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r777p\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-kube-api-access-r777p\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.576211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.576161 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-provider-credentials\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.576328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.576310 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-plugins-config-volume\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.576648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.576626 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-routing-overlay-volume\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.577266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.577238 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-plugins-config-volume\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.578705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.578685 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-provider-credentials\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.601217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.601178 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r777p\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-kube-api-access-r777p\") pod \"payload-processing-external-model-7bdf848cbc-5gjq2\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:33.687908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:33.687874 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:34.001322 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:34.001290 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47c742b_6317_448c_9238_ca761a18e273.slice/crio-ab6610999aa31fa2158217a7e3e33fb2c87e3570493d05e073dddee0bddd2b9a WatchSource:0}: Error finding container ab6610999aa31fa2158217a7e3e33fb2c87e3570493d05e073dddee0bddd2b9a: Status 404 returned error can't find the container with id ab6610999aa31fa2158217a7e3e33fb2c87e3570493d05e073dddee0bddd2b9a Nov 02 14:30:34.001526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:34.001500 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2"] Nov 02 14:30:34.077585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:34.076978 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:34.549761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:34.549717 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" event={"ID":"f47c742b-6317-448c-9238-ca761a18e273","Type":"ContainerStarted","Data":"cf45119813781a72f0ab835a2abf803810481e979e27edb57402c445a11c51e5"} Nov 02 14:30:34.550204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:34.549791 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" event={"ID":"f47c742b-6317-448c-9238-ca761a18e273","Type":"ContainerStarted","Data":"ab6610999aa31fa2158217a7e3e33fb2c87e3570493d05e073dddee0bddd2b9a"} Nov 02 14:30:34.550204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:34.549862 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:34.595288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:34.595216 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" podStartSLOduration=1.595196933 podStartE2EDuration="1.595196933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:34.591797238 +0000 UTC m=+2783.317419007" watchObservedRunningTime="2026-11-02 14:30:34.595196933 +0000 UTC m=+2783.320818704" Nov 02 14:30:35.697503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:35.697464 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l"] Nov 02 14:30:35.897301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:35.897266 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:36.165523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.165473 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" podUID="2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" containerName="payload-processing" containerID="cri-o://96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a" gracePeriod=25 Nov 02 14:30:36.452614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.452586 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:30:36.566093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.566060 5591 generic.go:356] "Generic (PLEG): container finished" podID="2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" containerID="96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a" exitCode=0 Nov 02 14:30:36.566277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.566111 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" Nov 02 14:30:36.566277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.566161 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" event={"ID":"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce","Type":"ContainerDied","Data":"96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a"} Nov 02 14:30:36.566277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.566223 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9" event={"ID":"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce","Type":"ContainerDied","Data":"96472b0fd00411f9771adf8adc9fdeae764c2d29e578a9b2ace4f06c13de5365"} Nov 02 14:30:36.566277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.566249 5591 scope.go:122] "RemoveContainer" containerID="96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a" Nov 02 14:30:36.578726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.578704 5591 scope.go:122] "RemoveContainer" containerID="96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a" Nov 02 14:30:36.579092 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:36.579064 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a\": container with ID starting with 96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a not found: ID does not exist" containerID="96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a" Nov 02 14:30:36.579228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.579101 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a"} err="failed to get container status \"96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a\": rpc error: code = NotFound desc = could not find container \"96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a\": container with ID starting with 96e52e75163e2069bd82e83dff186f839e8fa671d78030fc957e3af9fc07987a not found: ID does not exist" Nov 02 14:30:36.623640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.623612 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-routing-overlay-volume\") pod \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " Nov 02 14:30:36.623816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.623719 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-kube-api-access-jnp8l\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-kube-api-access-jnp8l\") pod \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " Nov 02 14:30:36.623816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.623808 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-provider-credentials\") pod \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " Nov 02 14:30:36.623962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.623866 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-plugins-config-volume\") pod \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\" (UID: \"2c48aaf7-0bed-4583-87c3-5f1af8cf98ce\") " Nov 02 14:30:36.624070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.623975 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-routing-overlay-volume" pod "2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" (UID: "2c48aaf7-0bed-4583-87c3-5f1af8cf98ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:36.624287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.624265 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-plugins-config-volume" pod "2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" (UID: "2c48aaf7-0bed-4583-87c3-5f1af8cf98ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:36.624423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.624404 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:36.624496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.624429 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:36.626033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.626007 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-kube-api-access-jnp8l" pod "2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" (UID: "2c48aaf7-0bed-4583-87c3-5f1af8cf98ce"). InnerVolumeSpecName "kube-api-access-jnp8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:36.626136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.626047 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-provider-credentials" pod "2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" (UID: "2c48aaf7-0bed-4583-87c3-5f1af8cf98ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:36.726030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.725926 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnp8l\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-kube-api-access-jnp8l\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:36.726030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.725961 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:36.927213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.927169 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9"] Nov 02 14:30:36.927431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:36.927242 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-655bbqvw9"] Nov 02 14:30:37.433542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:37.433503 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" path="/var/lib/kubelet/pods/2c48aaf7-0bed-4583-87c3-5f1af8cf98ce/volumes" Nov 02 14:30:37.590014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:37.584113 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:39.235504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:39.234250 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:40.698041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:40.697994 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" podUID="4d9408e7-09ae-4c84-82fc-58ec9f46f231" containerName="payload-processing" containerID="cri-o://e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312" gracePeriod=25 Nov 02 14:30:40.992995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:40.992953 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:30:41.070230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.070190 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:41.090363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.090324 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-plugins-config-volume\") pod \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " Nov 02 14:30:41.090591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.090561 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-provider-credentials\") pod \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " Nov 02 14:30:41.090679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.090633 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-kube-api-access-v4hh4\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-kube-api-access-v4hh4\") pod \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " Nov 02 14:30:41.090679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.090659 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-routing-overlay-volume\") pod \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\" (UID: \"4d9408e7-09ae-4c84-82fc-58ec9f46f231\") " Nov 02 14:30:41.090796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.090711 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-plugins-config-volume" pod "4d9408e7-09ae-4c84-82fc-58ec9f46f231" (UID: "4d9408e7-09ae-4c84-82fc-58ec9f46f231"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:41.091240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.091207 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-routing-overlay-volume" pod "4d9408e7-09ae-4c84-82fc-58ec9f46f231" (UID: "4d9408e7-09ae-4c84-82fc-58ec9f46f231"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:41.091468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.091446 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:41.091540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.091472 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d9408e7-09ae-4c84-82fc-58ec9f46f231-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:41.093052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.093023 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-kube-api-access-v4hh4" pod "4d9408e7-09ae-4c84-82fc-58ec9f46f231" (UID: "4d9408e7-09ae-4c84-82fc-58ec9f46f231"). InnerVolumeSpecName "kube-api-access-v4hh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:41.093270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.093250 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-provider-credentials" pod "4d9408e7-09ae-4c84-82fc-58ec9f46f231" (UID: "4d9408e7-09ae-4c84-82fc-58ec9f46f231"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:41.192787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.192745 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:41.192787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.192777 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4hh4\" (UniqueName: \"kubernetes.io/projected/4d9408e7-09ae-4c84-82fc-58ec9f46f231-kube-api-access-v4hh4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:41.618165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.618125 5591 generic.go:356] "Generic (PLEG): container finished" podID="4d9408e7-09ae-4c84-82fc-58ec9f46f231" containerID="e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312" exitCode=0 Nov 02 14:30:41.618370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.618170 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" event={"ID":"4d9408e7-09ae-4c84-82fc-58ec9f46f231","Type":"ContainerDied","Data":"e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312"} Nov 02 14:30:41.618370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.618216 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" event={"ID":"4d9408e7-09ae-4c84-82fc-58ec9f46f231","Type":"ContainerDied","Data":"23d2ba5225eaf78987b74ff5175707159e5231895a863e06492b93ebf61cf064"} Nov 02 14:30:41.618370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.618242 5591 scope.go:122] "RemoveContainer" containerID="e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312" Nov 02 14:30:41.618370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.618173 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l" Nov 02 14:30:41.628426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.628405 5591 scope.go:122] "RemoveContainer" containerID="e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312" Nov 02 14:30:41.628730 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:41.628708 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312\": container with ID starting with e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312 not found: ID does not exist" containerID="e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312" Nov 02 14:30:41.628803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.628738 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312"} err="failed to get container status \"e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312\": rpc error: code = NotFound desc = could not find container \"e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312\": container with ID starting with e32c109dddf6c9f514b140291301c95b2aecb9aaa8fc7bad4434e59bf15d8312 not found: ID does not exist" Nov 02 14:30:41.673770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.673739 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l"] Nov 02 14:30:41.686026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:41.685998 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7b7b8cs2l"] Nov 02 14:30:42.637008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:42.634832 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:43.431976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:43.431936 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9408e7-09ae-4c84-82fc-58ec9f46f231" path="/var/lib/kubelet/pods/4d9408e7-09ae-4c84-82fc-58ec9f46f231/volumes" Nov 02 14:30:44.388535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:44.388498 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:45.561155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:45.561124 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:30:45.704945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:45.704911 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm"] Nov 02 14:30:45.963666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:45.963629 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:46.129309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.129251 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:46.313668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.309402 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch"] Nov 02 14:30:46.318019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.315249 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:46.323342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.323309 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:46.323590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.323570 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:46.323725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.323712 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:46.323813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.323801 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7770092-321b-44d6-b18e-5f8268ecc9f9" containerName="cleanup" Nov 02 14:30:46.323899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.323888 5591 cpu_manager.go:414] "containerMap: removing container" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:46.323968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.323957 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:46.324126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324112 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" containerName="payload-processing" Nov 02 14:30:46.324208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324197 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" containerName="payload-processing" Nov 02 14:30:46.324291 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324282 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:46.324363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324354 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8659da97-3f58-48b2-84f1-a6d0edda1521" containerName="cleanup" Nov 02 14:30:46.324443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324433 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4d9408e7-09ae-4c84-82fc-58ec9f46f231" containerName="payload-processing" Nov 02 14:30:46.324508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324498 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9408e7-09ae-4c84-82fc-58ec9f46f231" containerName="payload-processing" Nov 02 14:30:46.324595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324586 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:46.324662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324652 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eeb0c5-2792-4561-b625-b0aaac536ea3" containerName="cleanup" Nov 02 14:30:46.324735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324727 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:46.324808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.324799 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c289825-cc5e-4c02-82ee-5e7eccafb3cc" containerName="cleanup" Nov 02 14:30:46.325674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.325104 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9239e8a6-5379-44a0-9588-385711ac8f51" containerName="cleanup" Nov 02 14:30:46.325848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.325831 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a71b7e-370a-4232-a9e5-f332a8fdb279" containerName="cleanup" Nov 02 14:30:46.325942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.325932 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7aeeb8-0306-4a22-8ef4-7376401f776c" containerName="cleanup" Nov 02 14:30:46.326047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.326035 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c48aaf7-0bed-4583-87c3-5f1af8cf98ce" containerName="payload-processing" Nov 02 14:30:46.326135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.326125 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f7180c-435b-4256-aa78-c27b8df3beab" containerName="cleanup" Nov 02 14:30:46.326223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.326214 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9408e7-09ae-4c84-82fc-58ec9f46f231" containerName="payload-processing" Nov 02 14:30:46.340005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.326293 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0154d150-d85a-4314-9df9-dbf526ada4b5" containerName="cleanup" Nov 02 14:30:46.354860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.354824 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch"] Nov 02 14:30:46.355299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.355200 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.371179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.371152 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-1665ff760a19\"" Nov 02 14:30:46.372082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.372061 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-e2e--7e93d834-dockercfg-vv77g\"" Nov 02 14:30:46.384273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.384242 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-plugins-e2e-extmod-a-26b508\"" Nov 02 14:30:46.384403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.384311 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"openshift-service-ca.crt\"" Nov 02 14:30:46.384403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.384315 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"routing-overlay\"" Nov 02 14:30:46.384403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.384379 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"kube-root-ca.crt\"" Nov 02 14:30:46.442680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.442636 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-kube-api-access-9jqsx\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.442886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.442735 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.442886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.442862 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.443084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.443066 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.544062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.544017 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.544273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.544079 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.544273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.544199 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.544403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.544307 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-kube-api-access-9jqsx\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.544846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.544820 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.544998 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.544940 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.546540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.546523 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.568426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.568365 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-kube-api-access-9jqsx\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.676876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.676845 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:46.981618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:46.981581 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch"] Nov 02 14:30:47.035445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.035417 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:47.058192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.058156 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch"] Nov 02 14:30:47.059307 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:47.059278 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15e10c9_e2d6_451c_bb94_939b4956023c.slice/crio-2ff390dad071d90bdb729fb28e87c5edf52f568812936e0658131e51faa0e58f WatchSource:0}: Error finding container 2ff390dad071d90bdb729fb28e87c5edf52f568812936e0658131e51faa0e58f: Status 404 returned error can't find the container with id 2ff390dad071d90bdb729fb28e87c5edf52f568812936e0658131e51faa0e58f Nov 02 14:30:47.208696 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.208657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:47.386094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.378936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:47.646920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.646808 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:47.699967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.699932 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" event={"ID":"d15e10c9-e2d6-451c-bb94-939b4956023c","Type":"ContainerStarted","Data":"07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1"} Nov 02 14:30:47.700157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.700005 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" event={"ID":"d15e10c9-e2d6-451c-bb94-939b4956023c","Type":"ContainerStarted","Data":"2ff390dad071d90bdb729fb28e87c5edf52f568812936e0658131e51faa0e58f"} Nov 02 14:30:47.700157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.700136 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:47.742953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:47.742906 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" podStartSLOduration=1.742894401 podStartE2EDuration="1.742894401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:47.74252191 +0000 UTC m=+2796.468143680" watchObservedRunningTime="2026-11-02 14:30:47.742894401 +0000 UTC m=+2796.468516172" Nov 02 14:30:48.204340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.204247 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:48.435767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.435733 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-ch89t"] Nov 02 14:30:48.440705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.440683 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.464173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.464109 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-ch89t"] Nov 02 14:30:48.514542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.514505 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-provider-credentials\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.514736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.514629 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-plugins-config-volume\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.514790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.514764 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-routing-overlay-volume\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.514953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.514932 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrr2f\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-kube-api-access-jrr2f\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.616576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.616544 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-plugins-config-volume\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.616770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.616611 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-routing-overlay-volume\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.616770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.616728 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrr2f\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-kube-api-access-jrr2f\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.616889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.616812 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-provider-credentials\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.617340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.617318 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-plugins-config-volume\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.617434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.617344 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-routing-overlay-volume\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.619376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.619356 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-provider-credentials\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.640628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.640600 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrr2f\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-kube-api-access-jrr2f\") pod \"payload-processing-external-model-56f5855db-ch89t\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:48.756500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:48.756422 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:49.068163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:49.068127 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-ch89t"] Nov 02 14:30:49.068308 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:49.068192 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d67fa7_b7ee_4625_9621_0135eac05eed.slice/crio-83464764280d04ad5f23e130acc9804116e4ac90c5391de8b860fef32a10bfc4 WatchSource:0}: Error finding container 83464764280d04ad5f23e130acc9804116e4ac90c5391de8b860fef32a10bfc4: Status 404 returned error can't find the container with id 83464764280d04ad5f23e130acc9804116e4ac90c5391de8b860fef32a10bfc4 Nov 02 14:30:49.496329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:49.496290 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:49.682852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:49.682820 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:49.775061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:49.774536 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" event={"ID":"78d67fa7-b7ee-4625-9621-0135eac05eed","Type":"ContainerStarted","Data":"141a902f0bfb4d347e293dcc78408acba0960cfe21b610fb6d4280b1c1fb9591"} Nov 02 14:30:49.775061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:49.774609 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" event={"ID":"78d67fa7-b7ee-4625-9621-0135eac05eed","Type":"ContainerStarted","Data":"83464764280d04ad5f23e130acc9804116e4ac90c5391de8b860fef32a10bfc4"} Nov 02 14:30:49.775061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:49.775027 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:49.827004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:49.826667 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" podStartSLOduration=1.826645823 podStartE2EDuration="1.826645823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:49.825387529 +0000 UTC m=+2798.551009302" watchObservedRunningTime="2026-11-02 14:30:49.826645823 +0000 UTC m=+2798.552267649" Nov 02 14:30:50.182284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:50.182235 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:50.705341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:50.705284 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" podUID="df9a965a-6c66-438d-a2f3-517472bdc7bc" containerName="payload-processing" containerID="cri-o://1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724" gracePeriod=25 Nov 02 14:30:50.985420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:50.985390 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:30:51.011153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.011124 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-kube-api-access-rcslz\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-kube-api-access-rcslz\") pod \"df9a965a-6c66-438d-a2f3-517472bdc7bc\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " Nov 02 14:30:51.011328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.011245 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-plugins-config-volume\") pod \"df9a965a-6c66-438d-a2f3-517472bdc7bc\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " Nov 02 14:30:51.011328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.011291 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-provider-credentials\") pod \"df9a965a-6c66-438d-a2f3-517472bdc7bc\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " Nov 02 14:30:51.011328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.011326 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-routing-overlay-volume\") pod \"df9a965a-6c66-438d-a2f3-517472bdc7bc\" (UID: \"df9a965a-6c66-438d-a2f3-517472bdc7bc\") " Nov 02 14:30:51.011746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.011712 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-plugins-config-volume" pod "df9a965a-6c66-438d-a2f3-517472bdc7bc" (UID: "df9a965a-6c66-438d-a2f3-517472bdc7bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:51.011887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.011821 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-routing-overlay-volume" pod "df9a965a-6c66-438d-a2f3-517472bdc7bc" (UID: "df9a965a-6c66-438d-a2f3-517472bdc7bc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:51.012161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.012140 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:51.012223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.012165 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9a965a-6c66-438d-a2f3-517472bdc7bc-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:51.013495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.013466 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-provider-credentials" pod "df9a965a-6c66-438d-a2f3-517472bdc7bc" (UID: "df9a965a-6c66-438d-a2f3-517472bdc7bc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:51.013587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.013498 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-kube-api-access-rcslz" pod "df9a965a-6c66-438d-a2f3-517472bdc7bc" (UID: "df9a965a-6c66-438d-a2f3-517472bdc7bc"). InnerVolumeSpecName "kube-api-access-rcslz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:51.113619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.113581 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcslz\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-kube-api-access-rcslz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:51.113619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.113612 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9a965a-6c66-438d-a2f3-517472bdc7bc-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:51.621258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.621219 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-ch89t"] Nov 02 14:30:51.662192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.662162 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f46955864-dhw9m"] Nov 02 14:30:51.662994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.662968 5591 cpu_manager.go:414] "containerMap: removing container" podUID="df9a965a-6c66-438d-a2f3-517472bdc7bc" containerName="payload-processing" Nov 02 14:30:51.663058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.662996 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9a965a-6c66-438d-a2f3-517472bdc7bc" containerName="payload-processing" Nov 02 14:30:51.663146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.663135 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9a965a-6c66-438d-a2f3-517472bdc7bc" containerName="payload-processing" Nov 02 14:30:51.667004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.666970 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.686179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.686144 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7d67e4e5499a\"" Nov 02 14:30:51.687749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.687720 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f46955864-dhw9m"] Nov 02 14:30:51.722346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.722260 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-plugins-config-volume\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.722499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.722437 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-provider-credentials\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.722572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.722510 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-routing-overlay-volume\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.722677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.722648 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmhh\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-kube-api-access-lwmhh\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.792620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.792582 5591 generic.go:356] "Generic (PLEG): container finished" podID="df9a965a-6c66-438d-a2f3-517472bdc7bc" containerID="1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724" exitCode=0 Nov 02 14:30:51.792840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.792633 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" Nov 02 14:30:51.792840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.792645 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" event={"ID":"df9a965a-6c66-438d-a2f3-517472bdc7bc","Type":"ContainerDied","Data":"1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724"} Nov 02 14:30:51.792840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.792712 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm" event={"ID":"df9a965a-6c66-438d-a2f3-517472bdc7bc","Type":"ContainerDied","Data":"6d66b5c7603ee0762dd0abfb425c59c6f4e8cd8f48ed8bc107dabde354efe040"} Nov 02 14:30:51.792840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.792736 5591 scope.go:122] "RemoveContainer" containerID="1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724" Nov 02 14:30:51.804835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.804815 5591 scope.go:122] "RemoveContainer" containerID="1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724" Nov 02 14:30:51.805151 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:51.805130 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724\": container with ID starting with 1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724 not found: ID does not exist" containerID="1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724" Nov 02 14:30:51.805213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.805158 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724"} err="failed to get container status \"1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724\": rpc error: code = NotFound desc = could not find container \"1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724\": container with ID starting with 1599875a8e2e1059de9e4fee4c03fa66796d6bd76b6b0d337f8190a225df2724 not found: ID does not exist" Nov 02 14:30:51.823827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.823801 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-plugins-config-volume\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.823950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.823929 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-provider-credentials\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.824049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.824032 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-routing-overlay-volume\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.824131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.824116 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmhh\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-kube-api-access-lwmhh\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.824644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.824621 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-plugins-config-volume\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.824644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.824634 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-routing-overlay-volume\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.826408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.826390 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-provider-credentials\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.911878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.911838 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmhh\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-kube-api-access-lwmhh\") pod \"payload-processing-external-model-f46955864-dhw9m\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.916141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.915056 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm"] Nov 02 14:30:51.918072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.918034 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8675f87ddd-rsmbm"] Nov 02 14:30:51.982237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.982140 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:51.985030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:51.985000 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:52.256933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.256852 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w"] Nov 02 14:30:52.263278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.263249 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.288148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.288114 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6"] Nov 02 14:30:52.293389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.293362 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.300959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.300780 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w"] Nov 02 14:30:52.307227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.307201 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f46955864-dhw9m"] Nov 02 14:30:52.313174 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:52.313139 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdb5ce1_98d8_4619_bc47_1ff390e0e9bf.slice/crio-662dbd4a02eeaf9d72e4f38a1f7d995cc675f8b1e7535995b370d5236ede7922 WatchSource:0}: Error finding container 662dbd4a02eeaf9d72e4f38a1f7d995cc675f8b1e7535995b370d5236ede7922: Status 404 returned error can't find the container with id 662dbd4a02eeaf9d72e4f38a1f7d995cc675f8b1e7535995b370d5236ede7922 Nov 02 14:30:52.320519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.320488 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6"] Nov 02 14:30:52.330305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330272 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.330412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330371 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eab821-d26d-4bde-addc-f46da6c42ac8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.330452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330415 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cef6fd-728f-4d68-a2d6-7818e861e12f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.330452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330445 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.330534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330474 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.330534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330517 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.330610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330551 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.330610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330578 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.330677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330643 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.330677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330666 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.330738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330711 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.330796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330778 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.330888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330861 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.330949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330908 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.331121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.330950 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.331178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.331146 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.331383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.331361 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfgb6\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-kube-api-access-xfgb6\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.331444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.331405 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qdck\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-kube-api-access-6qdck\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.437510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437465 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.437718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437540 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.437718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437585 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.437718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437633 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfgb6\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-kube-api-access-xfgb6\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.437718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437664 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdck\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-kube-api-access-6qdck\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.437958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437734 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.437958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437821 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eab821-d26d-4bde-addc-f46da6c42ac8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.437958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437864 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cef6fd-728f-4d68-a2d6-7818e861e12f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.437958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437895 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.437958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437928 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.438287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.437979 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.438287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.438039 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.438287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.438070 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.438287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.438151 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.438287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.438181 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.438287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.438233 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.438592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.438306 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.438592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.438382 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.438909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.438884 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.439193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.439168 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.439420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.439398 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.439652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.439611 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.439768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.439659 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.439871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.439846 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cef6fd-728f-4d68-a2d6-7818e861e12f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.439945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.439917 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.440219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.440196 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.440751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.440718 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eab821-d26d-4bde-addc-f46da6c42ac8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.441082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.441028 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.442187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.442160 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.442187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.442176 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.442716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.442691 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.442799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.442744 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.463481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.463451 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfgb6\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-kube-api-access-xfgb6\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.464772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.464753 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-rqcb6\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.467508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.467483 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdck\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-kube-api-access-6qdck\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.467588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.467550 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jw97w\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.585758 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.585660 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:52.630414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.630162 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:52.700659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.700615 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" podUID="d15e10c9-e2d6-451c-bb94-939b4956023c" containerName="payload-processing" containerID="cri-o://07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1" gracePeriod=25 Nov 02 14:30:52.804823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.804733 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" event={"ID":"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf","Type":"ContainerStarted","Data":"ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728"} Nov 02 14:30:52.805029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.804869 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" event={"ID":"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf","Type":"ContainerStarted","Data":"662dbd4a02eeaf9d72e4f38a1f7d995cc675f8b1e7535995b370d5236ede7922"} Nov 02 14:30:52.805029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.804975 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:30:52.854757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.854703 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" podStartSLOduration=1.854689217 podStartE2EDuration="1.854689217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:52.851257648 +0000 UTC m=+2801.576879419" watchObservedRunningTime="2026-11-02 14:30:52.854689217 +0000 UTC m=+2801.580311035" Nov 02 14:30:52.916138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.916094 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w"] Nov 02 14:30:52.920500 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:52.920467 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cef6fd_728f_4d68_a2d6_7818e861e12f.slice/crio-52d3601f829c3616e3d5a069fd8968f99badbed3f5660e18c285af62b1cb16e3 WatchSource:0}: Error finding container 52d3601f829c3616e3d5a069fd8968f99badbed3f5660e18c285af62b1cb16e3: Status 404 returned error can't find the container with id 52d3601f829c3616e3d5a069fd8968f99badbed3f5660e18c285af62b1cb16e3 Nov 02 14:30:52.923724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.923686 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:52.923830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.923758 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:52.923830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.923787 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:52.963900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.963857 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6"] Nov 02 14:30:52.968799 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:30:52.968762 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09eab821_d26d_4bde_addc_f46da6c42ac8.slice/crio-190f43f1ea0213f99f989b01ce2ffba6d546455ab3bb749dbc3fbb04999146d6 WatchSource:0}: Error finding container 190f43f1ea0213f99f989b01ce2ffba6d546455ab3bb749dbc3fbb04999146d6: Status 404 returned error can't find the container with id 190f43f1ea0213f99f989b01ce2ffba6d546455ab3bb749dbc3fbb04999146d6 Nov 02 14:30:52.972486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.972455 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:52.972562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.972527 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:52.972562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:52.972557 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:30:53.093254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.093226 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:53.163401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.162650 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-routing-overlay-volume\") pod \"d15e10c9-e2d6-451c-bb94-939b4956023c\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " Nov 02 14:30:53.163401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.162718 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-provider-credentials\") pod \"d15e10c9-e2d6-451c-bb94-939b4956023c\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " Nov 02 14:30:53.163401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.163016 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-plugins-config-volume\") pod \"d15e10c9-e2d6-451c-bb94-939b4956023c\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " Nov 02 14:30:53.163401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.163090 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-kube-api-access-9jqsx\") pod \"d15e10c9-e2d6-451c-bb94-939b4956023c\" (UID: \"d15e10c9-e2d6-451c-bb94-939b4956023c\") " Nov 02 14:30:53.167758 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.165356 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-plugins-config-volume" pod "d15e10c9-e2d6-451c-bb94-939b4956023c" (UID: "d15e10c9-e2d6-451c-bb94-939b4956023c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:53.167758 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.165930 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-routing-overlay-volume" pod "d15e10c9-e2d6-451c-bb94-939b4956023c" (UID: "d15e10c9-e2d6-451c-bb94-939b4956023c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:53.192174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.192126 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-kube-api-access-9jqsx" pod "d15e10c9-e2d6-451c-bb94-939b4956023c" (UID: "d15e10c9-e2d6-451c-bb94-939b4956023c"). InnerVolumeSpecName "kube-api-access-9jqsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:53.192358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.192190 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-provider-credentials" pod "d15e10c9-e2d6-451c-bb94-939b4956023c" (UID: "d15e10c9-e2d6-451c-bb94-939b4956023c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:53.264950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.264912 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:53.264950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.264950 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-kube-api-access-9jqsx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:53.265489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.264961 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d15e10c9-e2d6-451c-bb94-939b4956023c-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:53.265489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.264971 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d15e10c9-e2d6-451c-bb94-939b4956023c-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:53.434708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.434671 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9a965a-6c66-438d-a2f3-517472bdc7bc" path="/var/lib/kubelet/pods/df9a965a-6c66-438d-a2f3-517472bdc7bc/volumes" Nov 02 14:30:53.755515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.755384 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:53.835011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.831771 5591 generic.go:356] "Generic (PLEG): container finished" podID="d15e10c9-e2d6-451c-bb94-939b4956023c" containerID="07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1" exitCode=0 Nov 02 14:30:53.839013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.835286 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" Nov 02 14:30:53.847554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.835321 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" event={"ID":"d15e10c9-e2d6-451c-bb94-939b4956023c","Type":"ContainerDied","Data":"07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1"} Nov 02 14:30:53.848100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.848067 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch" event={"ID":"d15e10c9-e2d6-451c-bb94-939b4956023c","Type":"ContainerDied","Data":"2ff390dad071d90bdb729fb28e87c5edf52f568812936e0658131e51faa0e58f"} Nov 02 14:30:53.848261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.848246 5591 scope.go:122] "RemoveContainer" containerID="07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1" Nov 02 14:30:53.880016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.871257 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" event={"ID":"69cef6fd-728f-4d68-a2d6-7818e861e12f","Type":"ContainerStarted","Data":"d72b04daa3c99685f2aaad7b34bb4b2a2636f5de10adb64cd8913c64a47a2ca2"} Nov 02 14:30:53.880016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.871334 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" event={"ID":"69cef6fd-728f-4d68-a2d6-7818e861e12f","Type":"ContainerStarted","Data":"52d3601f829c3616e3d5a069fd8968f99badbed3f5660e18c285af62b1cb16e3"} Nov 02 14:30:53.902082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.902053 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" event={"ID":"09eab821-d26d-4bde-addc-f46da6c42ac8","Type":"ContainerStarted","Data":"a40bba18a070b32cea4624f3e8ac8ff2b206be8cf3365837ed635fe6f555dfee"} Nov 02 14:30:53.902196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.902102 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" event={"ID":"09eab821-d26d-4bde-addc-f46da6c42ac8","Type":"ContainerStarted","Data":"190f43f1ea0213f99f989b01ce2ffba6d546455ab3bb749dbc3fbb04999146d6"} Nov 02 14:30:53.902196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.902188 5591 scope.go:122] "RemoveContainer" containerID="07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1" Nov 02 14:30:53.902482 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:30:53.902461 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1\": container with ID starting with 07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1 not found: ID does not exist" containerID="07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1" Nov 02 14:30:53.902562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.902487 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1"} err="failed to get container status \"07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1\": rpc error: code = NotFound desc = could not find container \"07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1\": container with ID starting with 07d27c6290ac3a73cc70e787cbdb3e4da97fa280e1cc4521f9d55db4261019a1 not found: ID does not exist" Nov 02 14:30:53.929436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:53.929385 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podStartSLOduration=1.929371026 podStartE2EDuration="1.929371026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:53.924041261 +0000 UTC m=+2802.649663031" watchObservedRunningTime="2026-11-02 14:30:53.929371026 +0000 UTC m=+2802.654992797" Nov 02 14:30:54.009057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.008168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:54.017508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.012160 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch"] Nov 02 14:30:54.027397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.019802 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-cd96s9sch"] Nov 02 14:30:54.091650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.091583 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podStartSLOduration=2.091564278 podStartE2EDuration="2.091564278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:54.075338363 +0000 UTC m=+2802.800960134" watchObservedRunningTime="2026-11-02 14:30:54.091564278 +0000 UTC m=+2802.817186047" Nov 02 14:30:54.187729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.187690 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:54.587005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.586510 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:30:54.588281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.588260 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jw97w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:30:54.588349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.588304 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:30:54.635007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.634361 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:30:54.646007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.640124 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rqcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 02 14:30:54.646007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:54.640191 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 02 14:30:55.454008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:55.453414 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15e10c9-e2d6-451c-bb94-939b4956023c" path="/var/lib/kubelet/pods/d15e10c9-e2d6-451c-bb94-939b4956023c/volumes" Nov 02 14:30:55.564005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:55.563271 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:55.593006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:55.587832 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jw97w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:30:55.593006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:55.587906 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:30:55.640110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:55.639499 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rqcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 02 14:30:55.640110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:55.639567 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 02 14:30:56.084566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.084516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:56.256064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.256022 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:56.568811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.568774 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f46955864-dhw9m"] Nov 02 14:30:56.586953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.586911 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jw97w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:30:56.587198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.586975 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:30:56.630969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.630931 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rqcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 02 14:30:56.630969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.631024 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 02 14:30:56.793573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.793525 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" podUID="78d67fa7-b7ee-4625-9621-0135eac05eed" containerName="payload-processing" containerID="cri-o://141a902f0bfb4d347e293dcc78408acba0960cfe21b610fb6d4280b1c1fb9591" gracePeriod=25 Nov 02 14:30:56.973025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.972531 5591 generic.go:356] "Generic (PLEG): container finished" podID="78d67fa7-b7ee-4625-9621-0135eac05eed" containerID="141a902f0bfb4d347e293dcc78408acba0960cfe21b610fb6d4280b1c1fb9591" exitCode=0 Nov 02 14:30:56.973025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:56.972690 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" event={"ID":"78d67fa7-b7ee-4625-9621-0135eac05eed","Type":"ContainerDied","Data":"141a902f0bfb4d347e293dcc78408acba0960cfe21b610fb6d4280b1c1fb9591"} Nov 02 14:30:57.425753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.425721 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:57.502720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.502635 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-plugins-config-volume\") pod \"78d67fa7-b7ee-4625-9621-0135eac05eed\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " Nov 02 14:30:57.502908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.502842 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-provider-credentials\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-provider-credentials\") pod \"78d67fa7-b7ee-4625-9621-0135eac05eed\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " Nov 02 14:30:57.502908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.502899 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-kube-api-access-jrr2f\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-kube-api-access-jrr2f\") pod \"78d67fa7-b7ee-4625-9621-0135eac05eed\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " Nov 02 14:30:57.503084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.502959 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-routing-overlay-volume\") pod \"78d67fa7-b7ee-4625-9621-0135eac05eed\" (UID: \"78d67fa7-b7ee-4625-9621-0135eac05eed\") " Nov 02 14:30:57.504831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.504805 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-routing-overlay-volume" pod "78d67fa7-b7ee-4625-9621-0135eac05eed" (UID: "78d67fa7-b7ee-4625-9621-0135eac05eed"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:57.505121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.505093 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-plugins-config-volume" pod "78d67fa7-b7ee-4625-9621-0135eac05eed" (UID: "78d67fa7-b7ee-4625-9621-0135eac05eed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:57.506859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.506835 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-kube-api-access-jrr2f" pod "78d67fa7-b7ee-4625-9621-0135eac05eed" (UID: "78d67fa7-b7ee-4625-9621-0135eac05eed"). InnerVolumeSpecName "kube-api-access-jrr2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:57.507006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.506865 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-provider-credentials" pod "78d67fa7-b7ee-4625-9621-0135eac05eed" (UID: "78d67fa7-b7ee-4625-9621-0135eac05eed"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:57.586456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.586408 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jw97w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:30:57.586632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.586473 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:30:57.605740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.605703 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:57.605740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.605739 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrr2f\" (UniqueName: \"kubernetes.io/projected/78d67fa7-b7ee-4625-9621-0135eac05eed-kube-api-access-jrr2f\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:57.606127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.605756 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:57.606127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.605768 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d67fa7-b7ee-4625-9621-0135eac05eed-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:30:57.682012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.631375 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rqcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 02 14:30:57.685041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.682622 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 02 14:30:57.890844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.890813 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:57.989640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.989613 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" Nov 02 14:30:57.989640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.989631 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f5855db-ch89t" event={"ID":"78d67fa7-b7ee-4625-9621-0135eac05eed","Type":"ContainerDied","Data":"83464764280d04ad5f23e130acc9804116e4ac90c5391de8b860fef32a10bfc4"} Nov 02 14:30:57.989876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:57.989689 5591 scope.go:122] "RemoveContainer" containerID="141a902f0bfb4d347e293dcc78408acba0960cfe21b610fb6d4280b1c1fb9591" Nov 02 14:30:58.089102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:58.089061 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-ch89t"] Nov 02 14:30:58.094994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:58.094959 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-ch89t"] Nov 02 14:30:58.587034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:58.586826 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jw97w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:30:58.587034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:58.586892 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:30:58.653009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:58.630967 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rqcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 02 14:30:58.653009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:58.641195 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 02 14:30:58.898815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:58.898774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:59.168194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.168101 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:59.362028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.361326 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:59.458062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.455805 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d67fa7-b7ee-4625-9621-0135eac05eed" path="/var/lib/kubelet/pods/78d67fa7-b7ee-4625-9621-0135eac05eed/volumes" Nov 02 14:30:59.586373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.586329 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jw97w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:30:59.586583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.586403 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:30:59.594075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.594041 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:30:59.631063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.631031 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rqcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 02 14:30:59.631265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.631081 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 02 14:30:59.788601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:30:59.788513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:00.276856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:00.276818 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:00.515970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:00.515906 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:00.588634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:00.588544 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jw97w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:31:00.589149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:00.589112 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:31:00.631041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:00.631004 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rqcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body= Nov 02 14:31:00.631337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:00.631306 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" Nov 02 14:31:01.079950 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:31:01.079913 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:31:01.484003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:01.483352 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:01.575348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:01.575302 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" podUID="1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" containerName="payload-processing" containerID="cri-o://ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728" gracePeriod=25 Nov 02 14:31:01.692137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:01.692098 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:01.945998 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:01.945862 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:01.957092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:01.956730 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:31:02.047810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.046419 5591 generic.go:356] "Generic (PLEG): container finished" podID="1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" containerID="ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728" exitCode=0 Nov 02 14:31:02.047810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.046835 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" event={"ID":"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf","Type":"ContainerDied","Data":"ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728"} Nov 02 14:31:02.047810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.047002 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" event={"ID":"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf","Type":"ContainerDied","Data":"662dbd4a02eeaf9d72e4f38a1f7d995cc675f8b1e7535995b370d5236ede7922"} Nov 02 14:31:02.047810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.047054 5591 scope.go:122] "RemoveContainer" containerID="ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728" Nov 02 14:31:02.049104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.049040 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f46955864-dhw9m" Nov 02 14:31:02.050041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.049770 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-plugins-config-volume\") pod \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " Nov 02 14:31:02.050041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.049849 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-kube-api-access-lwmhh\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-kube-api-access-lwmhh\") pod \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " Nov 02 14:31:02.050041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.049893 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-routing-overlay-volume\") pod \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " Nov 02 14:31:02.050041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.049971 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-provider-credentials\") pod \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\" (UID: \"1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf\") " Nov 02 14:31:02.052204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.050342 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-plugins-config-volume" pod "1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" (UID: "1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:02.052204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.050628 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-routing-overlay-volume" pod "1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" (UID: "1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:02.057472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.057411 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-provider-credentials" pod "1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" (UID: "1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:02.063188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.063149 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-kube-api-access-lwmhh" pod "1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" (UID: "1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf"). InnerVolumeSpecName "kube-api-access-lwmhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:02.104842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.104814 5591 scope.go:122] "RemoveContainer" containerID="ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728" Nov 02 14:31:02.105874 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:31:02.105777 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728\": container with ID starting with ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728 not found: ID does not exist" containerID="ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728" Nov 02 14:31:02.105874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.105823 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728"} err="failed to get container status \"ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728\": rpc error: code = NotFound desc = could not find container \"ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728\": container with ID starting with ce79e10552c29885b7e2dfbb40a3834e9daca72328cc3ed17c726a8a90fac728 not found: ID does not exist" Nov 02 14:31:02.200006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.161497 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwmhh\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-kube-api-access-lwmhh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:02.201762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.200298 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:02.201762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.200334 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:02.201762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.200350 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:02.202620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.202593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.289867 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr"] Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291230 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" containerName="payload-processing" Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291255 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" containerName="payload-processing" Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291270 5591 cpu_manager.go:414] "containerMap: removing container" podUID="78d67fa7-b7ee-4625-9621-0135eac05eed" containerName="payload-processing" Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291279 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d67fa7-b7ee-4625-9621-0135eac05eed" containerName="payload-processing" Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291339 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d15e10c9-e2d6-451c-bb94-939b4956023c" containerName="payload-processing" Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291347 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15e10c9-e2d6-451c-bb94-939b4956023c" containerName="payload-processing" Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291533 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15e10c9-e2d6-451c-bb94-939b4956023c" containerName="payload-processing" Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291566 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" containerName="payload-processing" Nov 02 14:31:02.293033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.291578 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d67fa7-b7ee-4625-9621-0135eac05eed" containerName="payload-processing" Nov 02 14:31:02.319163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.314459 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.337741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.336485 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"payload-processing-external-model-e2e--aa97af6d-dockercfg-m4bd9\"" Nov 02 14:31:02.337741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.336559 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"agc-cred-0253073a10ea\"" Nov 02 14:31:02.340185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.340122 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ce0f55\"" Nov 02 14:31:02.340748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.340582 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"openshift-service-ca.crt\"" Nov 02 14:31:02.340748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.340658 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"routing-overlay\"" Nov 02 14:31:02.340748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.340721 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"kube-root-ca.crt\"" Nov 02 14:31:02.342058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.342035 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr"] Nov 02 14:31:02.438478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.438443 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:02.438924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.438758 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.438924 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.438867 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.439191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.439148 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.439348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.439323 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-kube-api-access-nmfnq\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.455753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.453782 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f46955864-dhw9m"] Nov 02 14:31:02.522009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.521944 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f46955864-dhw9m"] Nov 02 14:31:02.554239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.546485 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.554239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.548666 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.554239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.549077 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.554239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.549136 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.555968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.555939 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.556702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.556678 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-kube-api-access-nmfnq\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.563535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.563511 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.584603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.584479 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-kube-api-access-nmfnq\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.636005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.631362 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jw97w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:31:02.636005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.631466 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:02.659144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.659105 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:02.677180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.677142 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-rqcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:31:02.677461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:02.677433 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:03.072774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.072744 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:31:03.073684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.073665 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:31:03.075523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.075405 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:31:03.085730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.085699 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:03.109841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.109812 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr"] Nov 02 14:31:03.111180 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:31:03.111149 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5d2fc2_5a0c_42e9_9c5b_19a98ece3aa4.slice/crio-0900c66bebc7653cbe81e3349bf32d23d74c62dab415ad7d6b2c8f0c3d5d1580 WatchSource:0}: Error finding container 0900c66bebc7653cbe81e3349bf32d23d74c62dab415ad7d6b2c8f0c3d5d1580: Status 404 returned error can't find the container with id 0900c66bebc7653cbe81e3349bf32d23d74c62dab415ad7d6b2c8f0c3d5d1580 Nov 02 14:31:03.203461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.203431 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:31:03.204311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.204269 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:31:03.205063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.205035 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:31:03.335521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.335434 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:03.433695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:03.433659 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf" path="/var/lib/kubelet/pods/1fdb5ce1-98d8-4619-bc47-1ff390e0e9bf/volumes" Nov 02 14:31:04.156023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.145956 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" event={"ID":"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4","Type":"ContainerStarted","Data":"93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec"} Nov 02 14:31:04.156023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.147059 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:04.156023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.148653 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" event={"ID":"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4","Type":"ContainerStarted","Data":"0900c66bebc7653cbe81e3349bf32d23d74c62dab415ad7d6b2c8f0c3d5d1580"} Nov 02 14:31:04.204483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.204409 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" podStartSLOduration=2.204390317 podStartE2EDuration="2.204390317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:04.203655497 +0000 UTC m=+2812.929277267" watchObservedRunningTime="2026-11-02 14:31:04.204390317 +0000 UTC m=+2812.930012084" Nov 02 14:31:04.547820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.547728 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:04.730638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.730601 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:04.750240 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.750206 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk"] Nov 02 14:31:04.755442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.755419 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.781343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.781313 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk"] Nov 02 14:31:04.781487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.781363 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-97d52171139c\"" Nov 02 14:31:04.781487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.781444 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ee3eca\"" Nov 02 14:31:04.781628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.781610 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"kube-root-ca.crt\"" Nov 02 14:31:04.785192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.784803 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-e2e--6a9d33b0-dockercfg-5vjh6\"" Nov 02 14:31:04.785192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.784836 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"openshift-service-ca.crt\"" Nov 02 14:31:04.785192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.784959 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"routing-overlay\"" Nov 02 14:31:04.843943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.843852 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.844132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.844053 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.844197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.844181 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.844265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.844244 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqn8h\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-kube-api-access-cqn8h\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.964007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.963248 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.964007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.963418 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.964007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.963552 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.964007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.963598 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn8h\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-kube-api-access-cqn8h\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.986004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.964804 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.986004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.965555 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:04.986004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:04.971699 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:05.022014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:05.009540 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqn8h\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-kube-api-access-cqn8h\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:05.081244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:05.075662 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:05.421959 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:31:05.421916 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28440a84_4173_4977_9bcd_fd0739666992.slice/crio-d3c637c85cd2254c4e37e4b6e19ef96bf41072e2ddadfd685fd42d2302a03b52 WatchSource:0}: Error finding container d3c637c85cd2254c4e37e4b6e19ef96bf41072e2ddadfd685fd42d2302a03b52: Status 404 returned error can't find the container with id d3c637c85cd2254c4e37e4b6e19ef96bf41072e2ddadfd685fd42d2302a03b52 Nov 02 14:31:05.433544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:05.433514 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk"] Nov 02 14:31:05.627249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:05.627219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:06.219007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:06.218226 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" event={"ID":"28440a84-4173-4977-9bcd-fd0739666992","Type":"ContainerStarted","Data":"0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85"} Nov 02 14:31:06.219007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:06.218299 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:06.219007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:06.218324 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" event={"ID":"28440a84-4173-4977-9bcd-fd0739666992","Type":"ContainerStarted","Data":"d3c637c85cd2254c4e37e4b6e19ef96bf41072e2ddadfd685fd42d2302a03b52"} Nov 02 14:31:06.289048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:06.283059 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" podStartSLOduration=2.283036355 podStartE2EDuration="2.283036355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:06.264127453 +0000 UTC m=+2814.989749247" watchObservedRunningTime="2026-11-02 14:31:06.283036355 +0000 UTC m=+2815.008658127" Nov 02 14:31:06.370451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:06.370412 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:08.116007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:08.114653 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:08.950397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:08.950352 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm"] Nov 02 14:31:08.950610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:08.950583 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" containerID="cri-o://86b09d85e9e1ceb232ac994a2cc2f3b9947a5a956dc6ab1dcd27d8e1e8b80350" gracePeriod=30 Nov 02 14:31:09.101853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:09.101816 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2"] Nov 02 14:31:09.731905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:09.731201 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:11.404010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:11.403946 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:13.238259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:13.238217 5591 scope.go:122] "RemoveContainer" containerID="a580253a3f2037b7db0261713b5a20f3ffa760729e57f918814ab576c0119eca" Nov 02 14:31:13.290521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:13.289835 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:14.111064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:14.110061 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" podUID="f47c742b-6317-448c-9238-ca761a18e273" containerName="payload-processing" containerID="cri-o://cf45119813781a72f0ab835a2abf803810481e979e27edb57402c445a11c51e5" gracePeriod=25 Nov 02 14:31:14.563070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:14.560323 5591 generic.go:356] "Generic (PLEG): container finished" podID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerID="86b09d85e9e1ceb232ac994a2cc2f3b9947a5a956dc6ab1dcd27d8e1e8b80350" exitCode=0 Nov 02 14:31:14.563070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:14.560433 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" event={"ID":"781cfbc2-ec41-47df-9135-7a8454c039f7","Type":"ContainerDied","Data":"86b09d85e9e1ceb232ac994a2cc2f3b9947a5a956dc6ab1dcd27d8e1e8b80350"} Nov 02 14:31:14.582007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:14.566202 5591 generic.go:356] "Generic (PLEG): container finished" podID="f47c742b-6317-448c-9238-ca761a18e273" containerID="cf45119813781a72f0ab835a2abf803810481e979e27edb57402c445a11c51e5" exitCode=0 Nov 02 14:31:14.582500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:14.579069 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" event={"ID":"f47c742b-6317-448c-9238-ca761a18e273","Type":"ContainerDied","Data":"cf45119813781a72f0ab835a2abf803810481e979e27edb57402c445a11c51e5"} Nov 02 14:31:15.034024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.030079 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:15.530904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.530761 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:15.621622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.621585 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:15.633752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.628547 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:31:15.647223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.634385 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" event={"ID":"781cfbc2-ec41-47df-9135-7a8454c039f7","Type":"ContainerDied","Data":"1acd7f744c97d117c63568fcf4d59567dc8dc8a19a010ee7e906e1f45c7ad4a5"} Nov 02 14:31:15.647223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.634462 5591 scope.go:122] "RemoveContainer" containerID="86b09d85e9e1ceb232ac994a2cc2f3b9947a5a956dc6ab1dcd27d8e1e8b80350" Nov 02 14:31:15.657002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.656130 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.661800 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-certs\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.661888 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-credential-socket\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.661942 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-kube-api-access-h2r66\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-kube-api-access-h2r66\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662421 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-plugins-config-volume\") pod \"f47c742b-6317-448c-9238-ca761a18e273\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662520 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-token\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-token\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662558 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-podinfo\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662605 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-socket\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662685 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-envoy\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662766 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-kube-api-access-r777p\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-kube-api-access-r777p\") pod \"f47c742b-6317-448c-9238-ca761a18e273\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662840 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-routing-overlay-volume\") pod \"f47c742b-6317-448c-9238-ca761a18e273\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662872 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-data\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.665012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.662950 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-provider-credentials\") pod \"f47c742b-6317-448c-9238-ca761a18e273\" (UID: \"f47c742b-6317-448c-9238-ca761a18e273\") " Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.675889 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.677050 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-socket" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.677686 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-certs" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.677877 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-credential-socket" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.678041 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-podinfo" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.678147 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781cfbc2-ec41-47df-9135-7a8454c039f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cfbc2-ec41-47df-9135-7a8454c039f7-istiod-ca-cert\") pod \"781cfbc2-ec41-47df-9135-7a8454c039f7\" (UID: \"781cfbc2-ec41-47df-9135-7a8454c039f7\") " Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.679228 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.679252 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.679269 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.679285 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.681016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.680132 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-plugins-config-volume" pod "f47c742b-6317-448c-9238-ca761a18e273" (UID: "f47c742b-6317-448c-9238-ca761a18e273"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:15.687050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.681940 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-token" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:15.687050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.682413 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-data" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:15.687050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.682726 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781cfbc2-ec41-47df-9135-7a8454c039f7-istiod-ca-cert" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:15.693447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.693413 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-routing-overlay-volume" pod "f47c742b-6317-448c-9238-ca761a18e273" (UID: "f47c742b-6317-448c-9238-ca761a18e273"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:15.703011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.702204 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-kube-api-access-r777p" pod "f47c742b-6317-448c-9238-ca761a18e273" (UID: "f47c742b-6317-448c-9238-ca761a18e273"). InnerVolumeSpecName "kube-api-access-r777p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:15.703011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.702355 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-kube-api-access-h2r66" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "kube-api-access-h2r66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:15.706219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.706193 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-envoy" pod "781cfbc2-ec41-47df-9135-7a8454c039f7" (UID: "781cfbc2-ec41-47df-9135-7a8454c039f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:15.720445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.720412 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-provider-credentials" pod "f47c742b-6317-448c-9238-ca761a18e273" (UID: "f47c742b-6317-448c-9238-ca761a18e273"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:15.781493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.781399 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.781743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.781721 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r777p\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-kube-api-access-r777p\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.781860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.781845 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.781954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.781942 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.782110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.782092 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c742b-6317-448c-9238-ca761a18e273-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.782215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.782202 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cfbc2-ec41-47df-9135-7a8454c039f7-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.782313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.782301 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2r66\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-kube-api-access-h2r66\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.782413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.782400 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c742b-6317-448c-9238-ca761a18e273-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:15.782502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:15.782489 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cfbc2-ec41-47df-9135-7a8454c039f7-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:31:16.221557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.221526 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:31:16.558836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.558730 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" podUID="f47c742b-6317-448c-9238-ca761a18e273" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.234:50052\" within 1s: context deadline exceeded" Nov 02 14:31:16.558836 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:31:16.558829 5591 logging.go:55] [core] [Channel #7353 SubChannel #7354]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.234:50052", ServerName: "10.133.0.234:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.234:50052: operation was canceled" Nov 02 14:31:16.636178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.636148 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm" Nov 02 14:31:16.637719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.637696 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" Nov 02 14:31:16.637719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.637700 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2" event={"ID":"f47c742b-6317-448c-9238-ca761a18e273","Type":"ContainerDied","Data":"ab6610999aa31fa2158217a7e3e33fb2c87e3570493d05e073dddee0bddd2b9a"} Nov 02 14:31:16.637912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.637763 5591 scope.go:122] "RemoveContainer" containerID="cf45119813781a72f0ab835a2abf803810481e979e27edb57402c445a11c51e5" Nov 02 14:31:16.693129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.693093 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2"] Nov 02 14:31:16.697303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.697278 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bdf848cbc-5gjq2"] Nov 02 14:31:16.735098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.735064 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm"] Nov 02 14:31:16.738909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:16.738881 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-8s4wm"] Nov 02 14:31:17.068008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:17.066716 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:17.216119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:17.216090 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:31:17.434818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:17.434786 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" path="/var/lib/kubelet/pods/781cfbc2-ec41-47df-9135-7a8454c039f7/volumes" Nov 02 14:31:17.435378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:17.435358 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47c742b-6317-448c-9238-ca761a18e273" path="/var/lib/kubelet/pods/f47c742b-6317-448c-9238-ca761a18e273/volumes" Nov 02 14:31:18.960578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:18.960539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:20.616919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:20.616885 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:22.245730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:22.245681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:23.972009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:23.970900 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:25.474341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:25.474303 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:25.749453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:25.749332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:27.351695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:27.351625 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:29.050904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:29.050868 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:30.705904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:30.705868 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:31.076459 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:31:31.076373 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:31:32.382643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:32.382601 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:34.184584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:34.184529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:35.767067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:35.767027 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:37.381262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:37.381198 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:39.058013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:39.057923 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:40.784662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:40.784624 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:42.418666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:42.418585 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:44.079636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:44.079599 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:45.694255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:45.694200 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:47.353006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:47.352159 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:47.843965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:47.843927 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:31:49.165009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:49.162901 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:50.917571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:50.917533 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:52.661683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:52.661635 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:54.322234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:54.322195 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:55.932472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:55.932386 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:57.575874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:57.575810 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:31:59.264438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:31:59.264340 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:00.947643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:00.947586 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:01.077825 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:01.077783 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:32:02.665028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:02.664971 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:04.269534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:04.269494 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:05.866935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:05.866897 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:07.584017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:07.583265 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:09.259659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:09.259610 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:10.978490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:10.967598 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:12.565794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:12.565752 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:12.740006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:12.734712 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:13.003624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.003590 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2"] Nov 02 14:32:13.004366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.004351 5591 cpu_manager.go:414] "containerMap: removing container" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" Nov 02 14:32:13.004422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.004367 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" Nov 02 14:32:13.004422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.004395 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f47c742b-6317-448c-9238-ca761a18e273" containerName="payload-processing" Nov 02 14:32:13.004422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.004401 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47c742b-6317-448c-9238-ca761a18e273" containerName="payload-processing" Nov 02 14:32:13.004544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.004534 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="781cfbc2-ec41-47df-9135-7a8454c039f7" containerName="istio-proxy" Nov 02 14:32:13.004587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.004551 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47c742b-6317-448c-9238-ca761a18e273" containerName="payload-processing" Nov 02 14:32:13.008219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.008201 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.028061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.028027 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2"] Nov 02 14:32:13.037152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.037123 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"agc-cred-1247e009eddc\"" Nov 02 14:32:13.038161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.037447 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"openshift-service-ca.crt\"" Nov 02 14:32:13.038304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.037538 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"routing-overlay\"" Nov 02 14:32:13.038304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.038184 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"payload-processing-external-model-e2e--300d9f08-dockercfg-fg5j4\"" Nov 02 14:32:13.038304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.037700 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5f98a6\"" Nov 02 14:32:13.038304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.037602 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"kube-root-ca.crt\"" Nov 02 14:32:13.038701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.038673 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.038834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.038740 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.038904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.038846 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fktt\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-kube-api-access-5fktt\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.038960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.038923 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.140535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.140500 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fktt\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-kube-api-access-5fktt\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.140716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.140587 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.140716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.140713 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.140809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.140749 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.141487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.141459 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.141605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.141544 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.143228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.143203 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.164722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.164690 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fktt\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-kube-api-access-5fktt\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.323571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.323488 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:13.636554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.635681 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2"] Nov 02 14:32:13.636554 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:32:13.635931 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d56e0ef_8003_415d_aeb8_7e273a62b3e7.slice/crio-fd037a2c7a7694da75f47b32a22bc2eedecbd98eeac234c8c1aff80956dd7f46 WatchSource:0}: Error finding container fd037a2c7a7694da75f47b32a22bc2eedecbd98eeac234c8c1aff80956dd7f46: Status 404 returned error can't find the container with id fd037a2c7a7694da75f47b32a22bc2eedecbd98eeac234c8c1aff80956dd7f46 Nov 02 14:32:13.827461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.827427 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2"] Nov 02 14:32:13.982300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:13.982161 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:14.123234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:14.121201 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" event={"ID":"4d56e0ef-8003-415d-aeb8-7e273a62b3e7","Type":"ContainerStarted","Data":"00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42"} Nov 02 14:32:14.123234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:14.121276 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" event={"ID":"4d56e0ef-8003-415d-aeb8-7e273a62b3e7","Type":"ContainerStarted","Data":"fd037a2c7a7694da75f47b32a22bc2eedecbd98eeac234c8c1aff80956dd7f46"} Nov 02 14:32:14.123234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:14.121650 5591 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" secret="" err="secret \"payload-processing-external-model-e2e--300d9f08-dockercfg-fg5j4\" not found" Nov 02 14:32:14.123234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:14.122195 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:14.154720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:14.152556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:14.175422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:14.175352 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" podStartSLOduration=2.175335069 podStartE2EDuration="2.175335069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:14.172305173 +0000 UTC m=+2882.897926946" watchObservedRunningTime="2026-11-02 14:32:14.175335069 +0000 UTC m=+2882.900956841" Nov 02 14:32:14.290058 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.289488 5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-plugins-e2e-extmod-a-5f98a6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-5f98a6" not found Nov 02 14:32:14.290058 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.289592 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume podName:4d56e0ef-8003-415d-aeb8-7e273a62b3e7 nodeName:}" failed. No retries permitted until 2026-11-02 14:32:14.789565933 +0000 UTC m=+2883.515187701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-5f98a6" not found Nov 02 14:32:14.290058 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.289869 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5f98a6/agc-cred-1247e009eddc: secret "agc-cred-1247e009eddc" not found Nov 02 14:32:14.290058 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.289889 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2: secret "agc-cred-1247e009eddc" not found Nov 02 14:32:14.290058 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.289935 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials podName:4d56e0ef-8003-415d-aeb8-7e273a62b3e7 nodeName:}" failed. No retries permitted until 2026-11-02 14:32:14.789919216 +0000 UTC m=+2883.515541006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7") : secret "agc-cred-1247e009eddc" not found Nov 02 14:32:14.488910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:14.488582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:14.681728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:14.681693 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:14.822543 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.822506 5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-plugins-e2e-extmod-a-5f98a6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-5f98a6" not found Nov 02 14:32:14.822759 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.822598 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume podName:4d56e0ef-8003-415d-aeb8-7e273a62b3e7 nodeName:}" failed. No retries permitted until 2026-11-02 14:32:15.822573514 +0000 UTC m=+2884.548195281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-5f98a6" not found Nov 02 14:32:14.822759 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.822614 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5f98a6/agc-cred-1247e009eddc: secret "agc-cred-1247e009eddc" not found Nov 02 14:32:14.822759 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.822638 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2: secret "agc-cred-1247e009eddc" not found Nov 02 14:32:14.822759 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:14.822695 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials podName:4d56e0ef-8003-415d-aeb8-7e273a62b3e7 nodeName:}" failed. No retries permitted until 2026-11-02 14:32:15.822678438 +0000 UTC m=+2884.548300187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7") : secret "agc-cred-1247e009eddc" not found Nov 02 14:32:15.841910 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:15.841874 5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-plugins-e2e-extmod-a-5f98a6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-5f98a6" not found Nov 02 14:32:15.842444 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:15.841912 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5f98a6/agc-cred-1247e009eddc: secret "agc-cred-1247e009eddc" not found Nov 02 14:32:15.842444 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:15.841932 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2: secret "agc-cred-1247e009eddc" not found Nov 02 14:32:15.842444 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:15.841950 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume podName:4d56e0ef-8003-415d-aeb8-7e273a62b3e7 nodeName:}" failed. No retries permitted until 2026-11-02 14:32:17.84193479 +0000 UTC m=+2886.567556539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-5f98a6" not found Nov 02 14:32:15.842444 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:15.841969 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials podName:4d56e0ef-8003-415d-aeb8-7e273a62b3e7 nodeName:}" failed. No retries permitted until 2026-11-02 14:32:17.841958112 +0000 UTC m=+2886.567579860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7") : secret "agc-cred-1247e009eddc" not found Nov 02 14:32:16.381719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:16.381681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:17.872686 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:17.872626 5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-plugins-e2e-extmod-a-5f98a6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-5f98a6" not found Nov 02 14:32:17.873162 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:17.872719 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5f98a6/agc-cred-1247e009eddc: secret "agc-cred-1247e009eddc" not found Nov 02 14:32:17.873162 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:17.872736 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume podName:4d56e0ef-8003-415d-aeb8-7e273a62b3e7 nodeName:}" failed. No retries permitted until 2026-11-02 14:32:21.872715233 +0000 UTC m=+2890.598336999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-5f98a6" not found Nov 02 14:32:17.873162 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:17.872740 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2: secret "agc-cred-1247e009eddc" not found Nov 02 14:32:17.873162 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:17.872778 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials podName:4d56e0ef-8003-415d-aeb8-7e273a62b3e7 nodeName:}" failed. No retries permitted until 2026-11-02 14:32:21.872767783 +0000 UTC m=+2890.598389549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7") : secret "agc-cred-1247e009eddc" not found Nov 02 14:32:18.182407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:18.182261 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:19.920815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:19.920766 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:20.129311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.129265 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" podUID="4d56e0ef-8003-415d-aeb8-7e273a62b3e7" containerName="payload-processing" containerID="cri-o://00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42" gracePeriod=25 Nov 02 14:32:20.414308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.414279 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:20.421122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.421091 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-kube-api-access-5fktt\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-kube-api-access-5fktt\") pod \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " Nov 02 14:32:20.421272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.421168 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-routing-overlay-volume\") pod \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " Nov 02 14:32:20.421448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.421429 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials\") pod \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " Nov 02 14:32:20.421569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.421515 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume\") pod \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\" (UID: \"4d56e0ef-8003-415d-aeb8-7e273a62b3e7\") " Nov 02 14:32:20.421569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.421531 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-routing-overlay-volume" pod "4d56e0ef-8003-415d-aeb8-7e273a62b3e7" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:32:20.421955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.421925 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume" pod "4d56e0ef-8003-415d-aeb8-7e273a62b3e7" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:32:20.422373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.422348 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:32:20.422373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.422375 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:32:20.423539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.423513 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-kube-api-access-5fktt" pod "4d56e0ef-8003-415d-aeb8-7e273a62b3e7" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7"). InnerVolumeSpecName "kube-api-access-5fktt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:32:20.423829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.423815 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials" pod "4d56e0ef-8003-415d-aeb8-7e273a62b3e7" (UID: "4d56e0ef-8003-415d-aeb8-7e273a62b3e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:32:20.524713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.524613 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:32:20.524713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:20.524655 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fktt\" (UniqueName: \"kubernetes.io/projected/4d56e0ef-8003-415d-aeb8-7e273a62b3e7-kube-api-access-5fktt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:32:21.003915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.003790 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:21.183347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.183312 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:21.207559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.207524 5591 generic.go:356] "Generic (PLEG): container finished" podID="4d56e0ef-8003-415d-aeb8-7e273a62b3e7" containerID="00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42" exitCode=0 Nov 02 14:32:21.207743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.207581 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" Nov 02 14:32:21.207743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.207608 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" event={"ID":"4d56e0ef-8003-415d-aeb8-7e273a62b3e7","Type":"ContainerDied","Data":"00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42"} Nov 02 14:32:21.207743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.207662 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2" event={"ID":"4d56e0ef-8003-415d-aeb8-7e273a62b3e7","Type":"ContainerDied","Data":"fd037a2c7a7694da75f47b32a22bc2eedecbd98eeac234c8c1aff80956dd7f46"} Nov 02 14:32:21.207743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.207685 5591 scope.go:122] "RemoveContainer" containerID="00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42" Nov 02 14:32:21.221215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.221196 5591 scope.go:122] "RemoveContainer" containerID="00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42" Nov 02 14:32:21.221473 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:21.221452 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42\": container with ID starting with 00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42 not found: ID does not exist" containerID="00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42" Nov 02 14:32:21.221514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.221482 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42"} err="failed to get container status \"00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42\": rpc error: code = NotFound desc = could not find container \"00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42\": container with ID starting with 00f22cd7b39c915ac8d232c58e05ac6edf0b2f367f7234f4ec7f4b864079ea42 not found: ID does not exist" Nov 02 14:32:21.263770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.263681 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2"] Nov 02 14:32:21.276642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.276605 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc884pc2"] Nov 02 14:32:21.433955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.433910 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d56e0ef-8003-415d-aeb8-7e273a62b3e7" path="/var/lib/kubelet/pods/4d56e0ef-8003-415d-aeb8-7e273a62b3e7/volumes" Nov 02 14:32:21.582796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:21.582650 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:23.276473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:23.276369 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:24.746888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:24.746856 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:25.077242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:25.077149 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:25.602603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:25.602562 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:26.438953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:26.438921 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:32:26.878610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:26.878514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:28.853438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:28.853399 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:29.042588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:29.042525 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:29.341022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:29.338812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:29.632652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:29.632611 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:30.525048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:30.524795 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:30.702878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:30.702099 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:32:30.719490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:30.719083 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:32:30.754729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:30.754537 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:30.911172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:30.911123 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:32:30.957360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:30.957314 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:32:31.268560 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:32:31.268464 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:32:32.215542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.215510 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:32.731013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.730537 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:32.894703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.894671 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn"] Nov 02 14:32:32.895690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.895665 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4d56e0ef-8003-415d-aeb8-7e273a62b3e7" containerName="payload-processing" Nov 02 14:32:32.895690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.895688 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d56e0ef-8003-415d-aeb8-7e273a62b3e7" containerName="payload-processing" Nov 02 14:32:32.895890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.895847 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d56e0ef-8003-415d-aeb8-7e273a62b3e7" containerName="payload-processing" Nov 02 14:32:32.900953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.900930 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:32.920977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.920944 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a02c35\"" Nov 02 14:32:32.921141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.921026 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"agc-cred-8d633d498566\"" Nov 02 14:32:32.921141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.921046 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"kube-root-ca.crt\"" Nov 02 14:32:32.921141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.921098 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"routing-overlay\"" Nov 02 14:32:32.940224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.940184 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn"] Nov 02 14:32:32.952541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.952513 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:32.952687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.952590 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:32.952817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.952701 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4cl\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-kube-api-access-6k4cl\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:32.952886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:32.952815 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.021118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.021025 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"openshift-service-ca.crt\"" Nov 02 14:32:33.021304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.021195 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-e2e--ac2d6d70-dockercfg-sd6gb\"" Nov 02 14:32:33.069007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.064354 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.069007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.064456 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.069007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.064536 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4cl\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-kube-api-access-6k4cl\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.069007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.064639 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.069007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.065866 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.069460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.069145 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.105015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.097775 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.143006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.140710 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4cl\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-kube-api-access-6k4cl\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.219297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.219256 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.572313 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:32:33.572280 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903850ca_63e6_4183_afb0_c1562dc22fbe.slice/crio-a11bd8e1aeb8fd6a95d62388f40203509992a46b4599f075862f6ea2f5433ac1 WatchSource:0}: Error finding container a11bd8e1aeb8fd6a95d62388f40203509992a46b4599f075862f6ea2f5433ac1: Status 404 returned error can't find the container with id a11bd8e1aeb8fd6a95d62388f40203509992a46b4599f075862f6ea2f5433ac1 Nov 02 14:32:33.572644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.572617 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn"] Nov 02 14:32:33.684602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.684565 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" event={"ID":"903850ca-63e6-4183-afb0-c1562dc22fbe","Type":"ContainerStarted","Data":"20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32"} Nov 02 14:32:33.684793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.684628 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" event={"ID":"903850ca-63e6-4183-afb0-c1562dc22fbe","Type":"ContainerStarted","Data":"a11bd8e1aeb8fd6a95d62388f40203509992a46b4599f075862f6ea2f5433ac1"} Nov 02 14:32:33.684793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.684713 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:33.730103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.730045 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" podStartSLOduration=1.7300288190000002 podStartE2EDuration="1.730028819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:33.728640531 +0000 UTC m=+2902.454262298" watchObservedRunningTime="2026-11-02 14:32:33.730028819 +0000 UTC m=+2902.455650603" Nov 02 14:32:33.797432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:33.797400 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:34.005741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.005707 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn"] Nov 02 14:32:34.010413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.010394 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.038421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.038395 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"payload-processing-external-model-e2e--300d9f08-dockercfg-hnz47\"" Nov 02 14:32:34.038554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.038501 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"agc-cred-9322b40d138d\"" Nov 02 14:32:34.038882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.038865 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"openshift-service-ca.crt\"" Nov 02 14:32:34.038918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.038886 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"kube-root-ca.crt\"" Nov 02 14:32:34.038918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.038892 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"routing-overlay\"" Nov 02 14:32:34.039000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.038927 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5f98a6\"" Nov 02 14:32:34.090485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.090451 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.090645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.090630 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.090703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.090688 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph854\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-kube-api-access-ph854\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.090830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.090810 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.152242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.152160 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn"] Nov 02 14:32:34.349068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.347760 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.349068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.347871 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph854\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-kube-api-access-ph854\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.349068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.347976 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.349068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.348056 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.349745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.349550 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.362007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.351655 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.362007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.354277 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.457057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.451216 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:34.474538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.470849 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph854\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-kube-api-access-ph854\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.626659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.626578 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:34.977213 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:32:34.977176 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7493d54_d93f_4606_8c07_151d29accc7b.slice/crio-26c0447cc667f70c71d75f66e3133ca516334941d04c3068ac9e5c8f42ebd450 WatchSource:0}: Error finding container 26c0447cc667f70c71d75f66e3133ca516334941d04c3068ac9e5c8f42ebd450: Status 404 returned error can't find the container with id 26c0447cc667f70c71d75f66e3133ca516334941d04c3068ac9e5c8f42ebd450 Nov 02 14:32:34.977922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:34.977897 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn"] Nov 02 14:32:35.399846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:35.399803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:35.615912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:35.615842 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:35.740349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:35.740253 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" event={"ID":"d7493d54-d93f-4606-8c07-151d29accc7b","Type":"ContainerStarted","Data":"57ba70da5a3c7c5f2fb5dbfddbd5cafd4303a61455956a0406b943794875b553"} Nov 02 14:32:35.740629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:35.740600 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" event={"ID":"d7493d54-d93f-4606-8c07-151d29accc7b","Type":"ContainerStarted","Data":"26c0447cc667f70c71d75f66e3133ca516334941d04c3068ac9e5c8f42ebd450"} Nov 02 14:32:35.742125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:35.742058 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:35.803007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:35.798609 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" podStartSLOduration=2.798587801 podStartE2EDuration="2.798587801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:35.796491277 +0000 UTC m=+2904.522113048" watchObservedRunningTime="2026-11-02 14:32:35.798587801 +0000 UTC m=+2904.524209571" Nov 02 14:32:37.029528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.029493 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:37.292677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.292584 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:37.398788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.398749 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f"] Nov 02 14:32:37.404227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.404204 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.434964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.434932 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"agc-cred-9f3b5f00fe60\"" Nov 02 14:32:37.436245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.436201 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"routing-overlay\"" Nov 02 14:32:37.436676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.436505 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-66bc6d\"" Nov 02 14:32:37.436840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.436820 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"openshift-service-ca.crt\"" Nov 02 14:32:37.437498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.437475 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"kube-root-ca.crt\"" Nov 02 14:32:37.438215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.438192 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"payload-processing-external-model-e2e--47c96fd3-dockercfg-h4wx7\"" Nov 02 14:32:37.460371 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.460342 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f"] Nov 02 14:32:37.554731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.553008 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.554731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.553066 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7nw\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-kube-api-access-bs7nw\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.554731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.553136 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.554731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.553357 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.708009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.700572 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.708009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.700630 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7nw\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-kube-api-access-bs7nw\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.708009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.700699 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.708009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.700919 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.713005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.709794 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.713005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.710560 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.728128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.725164 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:37.734996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:37.734927 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7nw\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-kube-api-access-bs7nw\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:38.032746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:38.032712 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:38.342943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:38.342906 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f"] Nov 02 14:32:38.343195 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:32:38.343174 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bc298e_a73b_474d_9f0a_8302e39cef56.slice/crio-4b7b1124bf0e93275dce3de52011868588e48fc31ee4025182173669573ca9f4 WatchSource:0}: Error finding container 4b7b1124bf0e93275dce3de52011868588e48fc31ee4025182173669573ca9f4: Status 404 returned error can't find the container with id 4b7b1124bf0e93275dce3de52011868588e48fc31ee4025182173669573ca9f4 Nov 02 14:32:38.779603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:38.779531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:38.788618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:38.788571 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" event={"ID":"52bc298e-a73b-474d-9f0a-8302e39cef56","Type":"ContainerStarted","Data":"f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a"} Nov 02 14:32:38.788881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:38.788654 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" event={"ID":"52bc298e-a73b-474d-9f0a-8302e39cef56","Type":"ContainerStarted","Data":"4b7b1124bf0e93275dce3de52011868588e48fc31ee4025182173669573ca9f4"} Nov 02 14:32:38.788952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:38.788894 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:38.842152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:38.842091 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" podStartSLOduration=1.842075278 podStartE2EDuration="1.842075278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:38.840373682 +0000 UTC m=+2907.565995452" watchObservedRunningTime="2026-11-02 14:32:38.842075278 +0000 UTC m=+2907.567697048" Nov 02 14:32:40.483016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:40.473743 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:42.181044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:42.180777 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:43.869735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:43.869695 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:44.697632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:44.697596 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:32:45.464006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:45.452349 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:47.288458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:47.288416 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:47.539243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:47.539155 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:47.730669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:47.730595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:47.767338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:47.767306 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:32:49.258093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:49.254899 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:49.681973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:49.681940 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:49.799703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:49.799668 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:32:49.870003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:49.864392 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:51.164007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:51.163528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:52.848491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:52.848427 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:54.469561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:54.469521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:56.000704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:56.000671 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:56.171008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:56.168742 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:56.348379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:56.348302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:57.426547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:57.426514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:57.995367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:57.995327 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:32:59.771100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:32:59.771064 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:01.074995 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:33:01.074943 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:33:01.559101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:01.559054 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:02.133413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:02.133381 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:02.303319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:02.303281 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:03.472833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:03.472786 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:05.219543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:05.219502 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:06.979670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:06.979635 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:09.067014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:09.048468 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:09.233011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:09.225290 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:09.417009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:09.412931 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:10.687513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:10.687474 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:10.945066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:10.944028 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:11.156823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:11.156776 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:11.795009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:11.787571 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:11.957012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:11.949677 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:33:12.023205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:12.022351 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:33:12.069636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:12.068975 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 14:33:12.419009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:12.412461 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:33:12.847247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:12.847153 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:33:13.028832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:13.027297 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:13.258008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:13.254687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:14.028975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:14.028940 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:14.372432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:14.372394 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:15.184102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:15.184056 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:15.954636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:15.954593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:17.675152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:17.675113 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:18.557718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:18.557681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:18.723856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:18.723814 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:19.492922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:19.492866 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:21.165685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:21.165635 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:22.812576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:22.812505 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:23.836602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.836567 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj"] Nov 02 14:33:23.842373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.842337 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.858837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.858810 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5f98a6-openshift-default-dockercfg-4785q\"" Nov 02 14:33:23.863680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.863650 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-credential-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.863800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.863701 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istiod-ca-cert\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.863800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.863773 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-certs\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.863937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.863911 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-envoy\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.864045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.863999 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-podinfo\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.864173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.864153 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-token\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.864245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.864204 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd24n\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-kube-api-access-bd24n\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.864352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.864338 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.864451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.864438 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-data\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.875864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.875838 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj"] Nov 02 14:33:23.966142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966104 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-data\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966241 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-credential-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966275 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istiod-ca-cert\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966312 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-certs\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966352 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-envoy\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966387 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-podinfo\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966469 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-token\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966510 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd24n\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-kube-api-access-bd24n\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966583 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-data\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966600 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-credential-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966613 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966677 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-certs\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.966963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.966943 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.967076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.967052 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istiod-ca-cert\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.968731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.968702 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-envoy\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.969077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.969056 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-podinfo\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.991342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.991308 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-token\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:23.991466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:23.991427 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd24n\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-kube-api-access-bd24n\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:24.156811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:24.156767 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:24.437188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:24.432333 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:24.531937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:24.531902 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj"] Nov 02 14:33:24.533650 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:33:24.533618 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde3a69e_c4ea_48ec_b6ef_ceb1b1d7070b.slice/crio-98f0c6ac5bca515168184a9df2245e16710f425c7b798e3b44ef89a61b579e72 WatchSource:0}: Error finding container 98f0c6ac5bca515168184a9df2245e16710f425c7b798e3b44ef89a61b579e72: Status 404 returned error can't find the container with id 98f0c6ac5bca515168184a9df2245e16710f425c7b798e3b44ef89a61b579e72 Nov 02 14:33:24.536637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:24.536606 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:33:24.536723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:24.536673 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:33:24.536723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:24.536704 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:33:24.803129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:24.803048 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:25.087054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:25.086475 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:25.498654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:25.498613 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" event={"ID":"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b","Type":"ContainerStarted","Data":"0638431cf0826a37d3423daccd4c6cd8bbe7e70330e3fa525f4994e5d1792c15"} Nov 02 14:33:25.498854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:25.498672 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" event={"ID":"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b","Type":"ContainerStarted","Data":"98f0c6ac5bca515168184a9df2245e16710f425c7b798e3b44ef89a61b579e72"} Nov 02 14:33:25.557686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:25.557629 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" podStartSLOduration=2.557617741 podStartE2EDuration="2.557617741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:25.556739208 +0000 UTC m=+2954.282360978" watchObservedRunningTime="2026-11-02 14:33:25.557617741 +0000 UTC m=+2954.283239511" Nov 02 14:33:25.714914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:25.714882 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:26.160012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:26.157357 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:26.160012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:26.159255 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 02 14:33:26.160012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:26.159311 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 02 14:33:27.000366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:27.000286 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:27.157175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:27.157129 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 02 14:33:27.157385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:27.157193 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 02 14:33:28.158215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:28.158178 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 02 14:33:28.158635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:28.158237 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 02 14:33:28.858007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:28.856906 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:29.157553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:29.157514 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 02 14:33:29.157775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:29.157578 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 02 14:33:30.157678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:30.157630 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 02 14:33:30.158245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:30.157715 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 02 14:33:30.791786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:30.791749 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:31.085074 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:33:31.084972 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:33:31.157350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:31.157302 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Nov 02 14:33:31.157535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:31.157378 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Nov 02 14:33:32.222268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:32.222235 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:32.223884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:32.223859 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:32.224841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:32.224815 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:33:32.315525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:32.315488 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:32.728417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:32.728380 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:33:34.228644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:34.228563 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:36.163801 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:36.163757 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:36.862238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:36.862190 5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:33:36.958075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:36.958034 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:33:37.019007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:37.018087 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:33:38.000849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:38.000808 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:39.748384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:39.748342 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:41.444123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:41.442367 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:43.141025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:43.140943 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:44.563036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:44.563001 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:44.729083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:44.729049 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:44.903760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:44.903717 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:46.501006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:46.500208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:48.169089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:48.169030 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:49.776597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:49.776557 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:51.385005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:51.384306 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:53.078626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:53.070051 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:54.689661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:54.689621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:56.345255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:56.345216 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:57.968603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:57.968548 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:33:59.627004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:33:59.625242 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:01.076320 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:34:01.076277 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:34:01.278261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:01.273197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:02.974855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:02.974817 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:04.926930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:04.926801 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:07.081588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:07.081535 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:07.479110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:07.477077 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:34:07.696190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:07.696130 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:34:09.068062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:09.065388 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:11.045244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:11.045209 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:12.068534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:12.068507 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:34:12.097557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:12.097526 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:34:12.794008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:12.793504 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:14.471099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:14.471060 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:16.114452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:16.114415 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:17.764003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:17.763275 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:19.371605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:19.371566 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:21.034020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:21.033598 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:22.772008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:22.766828 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:23.021962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:23.021928 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:23.199335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:23.199298 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:24.681719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:24.681688 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:24.746281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:24.746239 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f"] Nov 02 14:34:24.887051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:24.887014 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:25.085534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.085438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:25.343103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.322190 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:25.578735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.578698 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2"] Nov 02 14:34:25.583455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.583438 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.617883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.617799 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-09d1a5-openshift-default-dockercfg-7r6q6\"" Nov 02 14:34:25.621976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.621940 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2"] Nov 02 14:34:25.730023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.728354 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:25.776377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.776338 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istiod-ca-cert\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.776783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.776734 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.777071 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.776979 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-data\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.777248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.777231 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-envoy\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.777488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.777470 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-podinfo\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.777666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.777650 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-certs\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.777953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.777935 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-credential-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.778228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.778208 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-token\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.778425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.778408 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6qd\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-kube-api-access-tp6qd\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.883365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.883329 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.883703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.883677 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-data\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.883881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.883865 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-envoy\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.884052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.884034 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-podinfo\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.884835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.884815 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-certs\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.889058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.889035 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-credential-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.896018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.891595 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-token\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.896792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.885234 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-certs\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.896971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.891127 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-credential-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.918315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.884748 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-data\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.918631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.884411 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.918770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.896822 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-podinfo\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.918896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.896760 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6qd\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-kube-api-access-tp6qd\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.919122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.919102 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istiod-ca-cert\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.920522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.920336 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istiod-ca-cert\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.925030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.922677 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:25.925030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.923149 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-envoy\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.952409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.952363 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6qd\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-kube-api-access-tp6qd\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:25.953769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:25.953737 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-token\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:26.128845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:26.128804 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:26.198020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:26.197891 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:26.524610 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:34:26.524580 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d71f880_d4ea_4d6d_9505_bccb5c82253b.slice/crio-58de4b51797b88a71a8b23c19d0ac654364754b024a47de58bfc6e138843ada4 WatchSource:0}: Error finding container 58de4b51797b88a71a8b23c19d0ac654364754b024a47de58bfc6e138843ada4: Status 404 returned error can't find the container with id 58de4b51797b88a71a8b23c19d0ac654364754b024a47de58bfc6e138843ada4 Nov 02 14:34:26.524740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:26.524717 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2"] Nov 02 14:34:26.527498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:26.527469 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:34:26.527587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:26.527534 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:34:26.527587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:26.527565 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:34:26.671020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:26.670871 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:27.314477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:27.314444 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:34:27.434863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:27.434824 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" event={"ID":"0d71f880-d4ea-4d6d-9505-bccb5c82253b","Type":"ContainerStarted","Data":"75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5"} Nov 02 14:34:27.435055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:27.434884 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" event={"ID":"0d71f880-d4ea-4d6d-9505-bccb5c82253b","Type":"ContainerStarted","Data":"58de4b51797b88a71a8b23c19d0ac654364754b024a47de58bfc6e138843ada4"} Nov 02 14:34:27.476477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:27.476427 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podStartSLOduration=2.476412982 podStartE2EDuration="2.476412982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:27.474956774 +0000 UTC m=+3016.200578543" watchObservedRunningTime="2026-11-02 14:34:27.476412982 +0000 UTC m=+3016.202034820" Nov 02 14:34:28.198262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:28.198222 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:28.205201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:28.205169 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:28.385600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:28.385520 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:28.433019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:28.432975 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:28.433822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:28.433802 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:34:29.746675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:29.746632 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" podUID="52bc298e-a73b-474d-9f0a-8302e39cef56" containerName="payload-processing" containerID="cri-o://f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a" gracePeriod=25 Nov 02 14:34:29.796772 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:34:29.796746 5591 logging.go:55] [core] [Channel #8615 SubChannel #8616]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.245:50052", ServerName: "10.133.0.245:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.245:50052: connect: connection refused" Nov 02 14:34:30.050111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.050070 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:34:30.091977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.091567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:30.243561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.243522 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-plugins-config-volume\") pod \"52bc298e-a73b-474d-9f0a-8302e39cef56\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " Nov 02 14:34:30.243849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.243820 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-kube-api-access-bs7nw\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-kube-api-access-bs7nw\") pod \"52bc298e-a73b-474d-9f0a-8302e39cef56\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " Nov 02 14:34:30.244773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.244750 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-routing-overlay-volume\") pod \"52bc298e-a73b-474d-9f0a-8302e39cef56\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " Nov 02 14:34:30.246065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.244373 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-plugins-config-volume" pod "52bc298e-a73b-474d-9f0a-8302e39cef56" (UID: "52bc298e-a73b-474d-9f0a-8302e39cef56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:34:30.246065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.245079 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-provider-credentials\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-provider-credentials\") pod \"52bc298e-a73b-474d-9f0a-8302e39cef56\" (UID: \"52bc298e-a73b-474d-9f0a-8302e39cef56\") " Nov 02 14:34:30.246065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.245435 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-routing-overlay-volume" pod "52bc298e-a73b-474d-9f0a-8302e39cef56" (UID: "52bc298e-a73b-474d-9f0a-8302e39cef56"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:34:30.256148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.246776 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:34:30.256148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.246814 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52bc298e-a73b-474d-9f0a-8302e39cef56-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:34:30.256148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.254517 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-kube-api-access-bs7nw" pod "52bc298e-a73b-474d-9f0a-8302e39cef56" (UID: "52bc298e-a73b-474d-9f0a-8302e39cef56"). InnerVolumeSpecName "kube-api-access-bs7nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:30.289151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.287298 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-provider-credentials" pod "52bc298e-a73b-474d-9f0a-8302e39cef56" (UID: "52bc298e-a73b-474d-9f0a-8302e39cef56"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:30.315971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.315894 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:30.357012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.353572 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs7nw\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-kube-api-access-bs7nw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:34:30.357012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.353611 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52bc298e-a73b-474d-9f0a-8302e39cef56-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:34:30.447890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.447849 5591 generic.go:356] "Generic (PLEG): container finished" podID="52bc298e-a73b-474d-9f0a-8302e39cef56" containerID="f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a" exitCode=0 Nov 02 14:34:30.448099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.447900 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" Nov 02 14:34:30.448099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.447923 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" event={"ID":"52bc298e-a73b-474d-9f0a-8302e39cef56","Type":"ContainerDied","Data":"f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a"} Nov 02 14:34:30.448099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.447970 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" event={"ID":"52bc298e-a73b-474d-9f0a-8302e39cef56","Type":"ContainerDied","Data":"4b7b1124bf0e93275dce3de52011868588e48fc31ee4025182173669573ca9f4"} Nov 02 14:34:30.448099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.448010 5591 scope.go:122] "RemoveContainer" containerID="f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a" Nov 02 14:34:30.461784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.461762 5591 scope.go:122] "RemoveContainer" containerID="f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a" Nov 02 14:34:30.462113 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:34:30.462092 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a\": container with ID starting with f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a not found: ID does not exist" containerID="f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a" Nov 02 14:34:30.462174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.462121 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a"} err="failed to get container status \"f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a\": rpc error: code = NotFound desc = could not find container \"f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a\": container with ID starting with f7389ed3c092b4974cef16e9483eeb47d496628ccec9db181014b5e0b624484a not found: ID does not exist" Nov 02 14:34:30.505601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.505569 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f"] Nov 02 14:34:30.518448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.518419 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f"] Nov 02 14:34:30.796972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:30.796931 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-84662fp6f" podUID="52bc298e-a73b-474d-9f0a-8302e39cef56" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.245:50052\" within 1s: context deadline exceeded" Nov 02 14:34:30.797424 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:34:30.796968 5591 logging.go:55] [core] [Channel #8615 SubChannel #8616]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.245:50052", ServerName: "10.133.0.245:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.245:50052: operation was canceled" Nov 02 14:34:31.075437 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:34:31.075344 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:34:31.435516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:31.435478 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bc298e-a73b-474d-9f0a-8302e39cef56" path="/var/lib/kubelet/pods/52bc298e-a73b-474d-9f0a-8302e39cef56/volumes" Nov 02 14:34:31.734104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:31.734022 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:31.901402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:31.901366 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:32.076063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:32.072527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:33.616897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:33.616855 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:35.366569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:35.366531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:37.069007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:37.065613 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:38.741389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:38.740529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:40.393002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:40.392947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:42.061797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:42.061753 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:43.664311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:43.664272 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:45.492116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:45.492074 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:47.158427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:47.158378 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:48.757154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:48.757113 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr"] Nov 02 14:34:48.812590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:48.812556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:48.978853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:48.978814 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:49.470481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:49.470443 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:50.585262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:50.585220 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:52.335635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:52.335599 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:53.758230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:53.758185 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" podUID="3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" containerName="payload-processing" containerID="cri-o://93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec" gracePeriod=25 Nov 02 14:34:53.952103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:53.951889 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:54.117675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.117640 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:34:54.166081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.166044 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-provider-credentials\") pod \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " Nov 02 14:34:54.166290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.166266 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-kube-api-access-nmfnq\") pod \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " Nov 02 14:34:54.166390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.166370 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-plugins-config-volume\") pod \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " Nov 02 14:34:54.166460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.166431 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-routing-overlay-volume\") pod \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\" (UID: \"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4\") " Nov 02 14:34:54.167515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.167484 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-routing-overlay-volume" pod "3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" (UID: "3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:34:54.167925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.167898 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-plugins-config-volume" pod "3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" (UID: "3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:34:54.169066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.169038 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-provider-credentials" pod "3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" (UID: "3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:54.169431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.169407 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-kube-api-access-nmfnq" pod "3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" (UID: "3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4"). InnerVolumeSpecName "kube-api-access-nmfnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:54.269192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.269153 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmfnq\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-kube-api-access-nmfnq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:34:54.269192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.269186 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:34:54.269192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.269196 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:34:54.269434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.269205 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:34:54.632951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.632912 5591 generic.go:356] "Generic (PLEG): container finished" podID="3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" containerID="93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec" exitCode=0 Nov 02 14:34:54.633173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.632961 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" Nov 02 14:34:54.633173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.633024 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" event={"ID":"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4","Type":"ContainerDied","Data":"93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec"} Nov 02 14:34:54.633173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.633079 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr" event={"ID":"3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4","Type":"ContainerDied","Data":"0900c66bebc7653cbe81e3349bf32d23d74c62dab415ad7d6b2c8f0c3d5d1580"} Nov 02 14:34:54.633173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.633101 5591 scope.go:122] "RemoveContainer" containerID="93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec" Nov 02 14:34:54.656594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.656571 5591 scope.go:122] "RemoveContainer" containerID="93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec" Nov 02 14:34:54.656921 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:34:54.656897 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec\": container with ID starting with 93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec not found: ID does not exist" containerID="93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec" Nov 02 14:34:54.657965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.656928 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec"} err="failed to get container status \"93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec\": rpc error: code = NotFound desc = could not find container \"93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec\": container with ID starting with 93515e7a4f7264f96c25b5436b2d76b0d2a3c9a9c93143a8bc3428d0e492a4ec not found: ID does not exist" Nov 02 14:34:54.689633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.689595 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr"] Nov 02 14:34:54.701849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:54.701820 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-6dc4bgqnr"] Nov 02 14:34:55.435187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:55.435153 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" path="/var/lib/kubelet/pods/3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4/volumes" Nov 02 14:34:55.616473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:55.616332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:55.790704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:55.790601 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:55.990299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:55.990259 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:56.234478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:56.234431 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:56.555226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:56.555081 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:57.219159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:57.219093 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:57.487060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:57.482503 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:57.724579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:57.724532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:34:59.638836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:34:59.615039 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:00.239021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.231197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:00.635905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.635847 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:00.832219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.832179 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:00.848020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.846597 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh"] Nov 02 14:35:00.848020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.847873 5591 cpu_manager.go:414] "containerMap: removing container" podUID="52bc298e-a73b-474d-9f0a-8302e39cef56" containerName="payload-processing" Nov 02 14:35:00.848020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.847900 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bc298e-a73b-474d-9f0a-8302e39cef56" containerName="payload-processing" Nov 02 14:35:00.848020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.847933 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" containerName="payload-processing" Nov 02 14:35:00.848020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.847941 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" containerName="payload-processing" Nov 02 14:35:00.848508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.848124 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bc298e-a73b-474d-9f0a-8302e39cef56" containerName="payload-processing" Nov 02 14:35:00.848508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.848144 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5d2fc2-5a0c-42e9-9c5b-19a98ece3aa4" containerName="payload-processing" Nov 02 14:35:00.912059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.893356 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh"] Nov 02 14:35:00.912059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.893493 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:00.920082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.920049 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"agc-cred-ded744473122\"" Nov 02 14:35:00.960007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.934418 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"kube-root-ca.crt\"" Nov 02 14:35:00.960007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.934565 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"payload-processing-external-model-e2e--aa97af6d-dockercfg-7n495\"" Nov 02 14:35:00.960007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.934687 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ce0f55\"" Nov 02 14:35:00.960007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.934754 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"openshift-service-ca.crt\"" Nov 02 14:35:00.960007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:00.934852 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"routing-overlay\"" Nov 02 14:35:01.013008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.006597 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.013008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.006680 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.013008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.006710 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.013008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.006764 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pg82\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-kube-api-access-2pg82\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.263065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.173761 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.263065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.173850 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.263065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.173888 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.263065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.173951 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pg82\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-kube-api-access-2pg82\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.263065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.175291 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.263065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.176032 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.522130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.518688 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.550557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.529721 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pg82\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-kube-api-access-2pg82\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:01.648712 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:35:01.648672 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:35:01.697302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:01.697263 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:02.304009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:02.302309 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:02.612213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:02.612174 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh"] Nov 02 14:35:02.623914 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:35:02.623880 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521fbf4d_dc67_43be_a448_b33b1190ef5c.slice/crio-ca7397843de52ec5fc3a5357a83fc932c698995fea708c87ad1e1d8ed7a08a5e WatchSource:0}: Error finding container ca7397843de52ec5fc3a5357a83fc932c698995fea708c87ad1e1d8ed7a08a5e: Status 404 returned error can't find the container with id ca7397843de52ec5fc3a5357a83fc932c698995fea708c87ad1e1d8ed7a08a5e Nov 02 14:35:02.876403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:02.876297 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-77nsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:35:02.876695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:02.876645 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:35:02.918563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:02.917211 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" event={"ID":"521fbf4d-dc67-43be-a448-b33b1190ef5c","Type":"ContainerStarted","Data":"a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b"} Nov 02 14:35:02.918563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:02.917292 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" event={"ID":"521fbf4d-dc67-43be-a448-b33b1190ef5c","Type":"ContainerStarted","Data":"ca7397843de52ec5fc3a5357a83fc932c698995fea708c87ad1e1d8ed7a08a5e"} Nov 02 14:35:02.918563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:02.917327 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:02.970007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:02.969571 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" podStartSLOduration=2.9695514469999997 podStartE2EDuration="2.969551447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:02.966507382 +0000 UTC m=+3051.692129156" watchObservedRunningTime="2026-11-02 14:35:02.969551447 +0000 UTC m=+3051.695173218" Nov 02 14:35:04.271810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:04.271771 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:04.973649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:04.973616 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh"] Nov 02 14:35:05.133974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:05.133936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:05.314635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:05.314549 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:05.794918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:05.794885 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:05.971516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:05.971481 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:06.442288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:06.442249 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:06.965257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:06.965219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:06.977012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:06.976796 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq"] Nov 02 14:35:07.030083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:06.998484 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.030083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.011580 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq"] Nov 02 14:35:07.048051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.045498 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-a6650dd4fae6\"" Nov 02 14:35:07.048051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.046156 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-18bea525a9f6\"" Nov 02 14:35:07.058761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.058568 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-plugins-e2e-extmod-a-26b508\"" Nov 02 14:35:07.059501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.059450 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"openshift-service-ca.crt\"" Nov 02 14:35:07.059771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.059742 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"routing-overlay\"" Nov 02 14:35:07.060048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.060018 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"kube-root-ca.crt\"" Nov 02 14:35:07.065188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.065155 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-e2e--7e93d834-dockercfg-gcw7q\"" Nov 02 14:35:07.141919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.141887 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:07.155754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.155716 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.155916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.155845 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.156089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.156075 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.156197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.156095 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-kube-api-access-rgggq\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.258343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.258240 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.258343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.258288 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-kube-api-access-rgggq\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.258600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.258425 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.258600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.258486 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.259086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.259064 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.259210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.259189 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.261025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.261005 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.283567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.283533 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-kube-api-access-rgggq\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.370614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.370576 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:07.689905 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:35:07.689874 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479119f4_57dc_429c_aee8_69dadd26647a.slice/crio-2755a8fa2bf8ba896d6c9ba665f41bad097a2cc0f0104233ab214098e765720a WatchSource:0}: Error finding container 2755a8fa2bf8ba896d6c9ba665f41bad097a2cc0f0104233ab214098e765720a: Status 404 returned error can't find the container with id 2755a8fa2bf8ba896d6c9ba665f41bad097a2cc0f0104233ab214098e765720a Nov 02 14:35:07.690256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.690195 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq"] Nov 02 14:35:07.777583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.777557 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6"] Nov 02 14:35:07.782108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.782090 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:07.798203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.798160 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-17bd01618a1c\"" Nov 02 14:35:07.802162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.802133 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6"] Nov 02 14:35:07.972349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.972258 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:07.972349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.972325 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:07.972583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.972558 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:07.972710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:07.972697 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8nd\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-kube-api-access-2l8nd\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.061215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.061166 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" event={"ID":"479119f4-57dc-429c-aee8-69dadd26647a","Type":"ContainerStarted","Data":"83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0"} Nov 02 14:35:08.061404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.061263 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" event={"ID":"479119f4-57dc-429c-aee8-69dadd26647a","Type":"ContainerStarted","Data":"2755a8fa2bf8ba896d6c9ba665f41bad097a2cc0f0104233ab214098e765720a"} Nov 02 14:35:08.061484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.061414 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:08.073612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.073589 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.073750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.073638 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.073809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.073775 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.074237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.074185 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8nd\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-kube-api-access-2l8nd\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.074524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.074493 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.074744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.074714 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.076249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.076225 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.098322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.098293 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8nd\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-kube-api-access-2l8nd\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.107944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.107902 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" podStartSLOduration=2.107890423 podStartE2EDuration="2.107890423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:08.103880692 +0000 UTC m=+3056.829502461" watchObservedRunningTime="2026-11-02 14:35:08.107890423 +0000 UTC m=+3056.833512193" Nov 02 14:35:08.277060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.273947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:08.396919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.396890 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:08.443849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.443811 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:08.846485 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:35:08.846451 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dc64d7_c8ee_4a2f_a154_b3ce1d027005.slice/crio-2a4f58cee3b500740410bca839ac4fd5a5b13793588d7e986f556870b26bc32c WatchSource:0}: Error finding container 2a4f58cee3b500740410bca839ac4fd5a5b13793588d7e986f556870b26bc32c: Status 404 returned error can't find the container with id 2a4f58cee3b500740410bca839ac4fd5a5b13793588d7e986f556870b26bc32c Nov 02 14:35:08.846926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:08.846508 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6"] Nov 02 14:35:09.070187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:09.070148 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" event={"ID":"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005","Type":"ContainerStarted","Data":"3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93"} Nov 02 14:35:09.070379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:09.070210 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" event={"ID":"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005","Type":"ContainerStarted","Data":"2a4f58cee3b500740410bca839ac4fd5a5b13793588d7e986f556870b26bc32c"} Nov 02 14:35:09.070476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:09.070433 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:09.118076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:09.117957 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" podStartSLOduration=2.117942683 podStartE2EDuration="2.117942683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:09.115653079 +0000 UTC m=+3057.841274844" watchObservedRunningTime="2026-11-02 14:35:09.117942683 +0000 UTC m=+3057.843564452" Nov 02 14:35:09.974218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:09.974178 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" podUID="521fbf4d-dc67-43be-a448-b33b1190ef5c" containerName="payload-processing" containerID="cri-o://a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b" gracePeriod=25 Nov 02 14:35:10.041678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.041495 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:10.270238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.270212 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:10.359906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.359866 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-plugins-config-volume\") pod \"521fbf4d-dc67-43be-a448-b33b1190ef5c\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " Nov 02 14:35:10.360129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.360108 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-routing-overlay-volume\") pod \"521fbf4d-dc67-43be-a448-b33b1190ef5c\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " Nov 02 14:35:10.360329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.360312 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-kube-api-access-2pg82\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-kube-api-access-2pg82\") pod \"521fbf4d-dc67-43be-a448-b33b1190ef5c\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " Nov 02 14:35:10.360385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.360358 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-provider-credentials\") pod \"521fbf4d-dc67-43be-a448-b33b1190ef5c\" (UID: \"521fbf4d-dc67-43be-a448-b33b1190ef5c\") " Nov 02 14:35:10.360435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.360410 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-plugins-config-volume" pod "521fbf4d-dc67-43be-a448-b33b1190ef5c" (UID: "521fbf4d-dc67-43be-a448-b33b1190ef5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:10.360474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.360444 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-routing-overlay-volume" pod "521fbf4d-dc67-43be-a448-b33b1190ef5c" (UID: "521fbf4d-dc67-43be-a448-b33b1190ef5c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:10.361089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.361062 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:10.361089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.361090 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521fbf4d-dc67-43be-a448-b33b1190ef5c-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:10.363193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.363172 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-kube-api-access-2pg82" pod "521fbf4d-dc67-43be-a448-b33b1190ef5c" (UID: "521fbf4d-dc67-43be-a448-b33b1190ef5c"). InnerVolumeSpecName "kube-api-access-2pg82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:10.363272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.363201 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-provider-credentials" pod "521fbf4d-dc67-43be-a448-b33b1190ef5c" (UID: "521fbf4d-dc67-43be-a448-b33b1190ef5c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:10.462784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.462743 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pg82\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-kube-api-access-2pg82\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:10.462784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:10.462775 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/521fbf4d-dc67-43be-a448-b33b1190ef5c-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:11.096343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.096304 5591 generic.go:356] "Generic (PLEG): container finished" podID="521fbf4d-dc67-43be-a448-b33b1190ef5c" containerID="a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b" exitCode=0 Nov 02 14:35:11.096343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.096354 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" Nov 02 14:35:11.096815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.096391 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" event={"ID":"521fbf4d-dc67-43be-a448-b33b1190ef5c","Type":"ContainerDied","Data":"a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b"} Nov 02 14:35:11.096815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.096454 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh" event={"ID":"521fbf4d-dc67-43be-a448-b33b1190ef5c","Type":"ContainerDied","Data":"ca7397843de52ec5fc3a5357a83fc932c698995fea708c87ad1e1d8ed7a08a5e"} Nov 02 14:35:11.096815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.096481 5591 scope.go:122] "RemoveContainer" containerID="a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b" Nov 02 14:35:11.107511 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.107491 5591 scope.go:122] "RemoveContainer" containerID="a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b" Nov 02 14:35:11.107794 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:35:11.107776 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b\": container with ID starting with a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b not found: ID does not exist" containerID="a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b" Nov 02 14:35:11.107858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.107806 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b"} err="failed to get container status \"a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b\": rpc error: code = NotFound desc = could not find container \"a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b\": container with ID starting with a5eb4fbf29cfdcba7362c3635b1cb573e9a4bffdab97dc8b710af629ee5a1a7b not found: ID does not exist" Nov 02 14:35:11.171638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.171597 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh"] Nov 02 14:35:11.184179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.184152 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5579d74jh"] Nov 02 14:35:11.453868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.453820 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521fbf4d-dc67-43be-a448-b33b1190ef5c" path="/var/lib/kubelet/pods/521fbf4d-dc67-43be-a448-b33b1190ef5c/volumes" Nov 02 14:35:11.697680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:11.697580 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:12.817507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:12.817438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:13.005097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:13.004997 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:13.289969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:13.289918 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:13.476849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:13.476814 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:13.827012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:13.814169 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:14.218394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:14.218358 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:15.332012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:15.331509 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:17.295289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:17.295254 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:17.461033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:17.460947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:17.743785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:17.743751 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:17.913088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:17.913050 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:19.074448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:19.074417 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:35:19.177186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:19.177147 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:20.088774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.088744 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:35:20.192046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.192004 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk"] Nov 02 14:35:20.219724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.219675 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:20.315682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.315641 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n"] Nov 02 14:35:20.316517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.316492 5591 cpu_manager.go:414] "containerMap: removing container" podUID="521fbf4d-dc67-43be-a448-b33b1190ef5c" containerName="payload-processing" Nov 02 14:35:20.316517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.316511 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521fbf4d-dc67-43be-a448-b33b1190ef5c" containerName="payload-processing" Nov 02 14:35:20.316654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.316632 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="521fbf4d-dc67-43be-a448-b33b1190ef5c" containerName="payload-processing" Nov 02 14:35:20.321894 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.321876 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:20.379756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.379666 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e1fcb4\"" Nov 02 14:35:20.380780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.380730 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e1fcb4-dockercfg-c9jt8\"" Nov 02 14:35:20.392789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.392750 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n"] Nov 02 14:35:20.515375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.509570 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvw7n\" (UniqueName: \"kubernetes.io/projected/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-kube-api-access-xvw7n\") pod \"payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n\" (UID: \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:20.515375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.509627 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n\" (UID: \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:20.544571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.522519 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:20.642008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.635351 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvw7n\" (UniqueName: \"kubernetes.io/projected/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-kube-api-access-xvw7n\") pod \"payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n\" (UID: \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:20.642008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.635416 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n\" (UID: \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:20.643897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.643313 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n\" (UID: \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:20.745843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.745803 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvw7n\" (UniqueName: \"kubernetes.io/projected/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-kube-api-access-xvw7n\") pod \"payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n\" (UID: \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:20.936886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:20.936806 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:21.146010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:21.144911 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:21.253215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:21.253189 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n"] Nov 02 14:35:21.253643 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:35:21.253623 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebeafe1a_8227_4ab6_9cd7_9f5f3ba75d2c.slice/crio-9fc82279277dcd0ce5b7bc4c2ae8ecd3941a5bb74d82684f3bbe1373a8fb57e9 WatchSource:0}: Error finding container 9fc82279277dcd0ce5b7bc4c2ae8ecd3941a5bb74d82684f3bbe1373a8fb57e9: Status 404 returned error can't find the container with id 9fc82279277dcd0ce5b7bc4c2ae8ecd3941a5bb74d82684f3bbe1373a8fb57e9 Nov 02 14:35:22.190705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:22.190661 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" event={"ID":"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c","Type":"ContainerStarted","Data":"0235655b7fa965f77c7fb07989a608883a5770a333ca4f2a0e4c5cf29860d27d"} Nov 02 14:35:22.191148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:22.190733 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:22.191148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:22.190754 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" event={"ID":"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c","Type":"ContainerStarted","Data":"9fc82279277dcd0ce5b7bc4c2ae8ecd3941a5bb74d82684f3bbe1373a8fb57e9"} Nov 02 14:35:22.238367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:22.238309 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" podStartSLOduration=2.238295535 podStartE2EDuration="2.238295535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:22.235308094 +0000 UTC m=+3070.960929865" watchObservedRunningTime="2026-11-02 14:35:22.238295535 +0000 UTC m=+3070.963917304" Nov 02 14:35:22.955068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:22.948125 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:25.053772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.053741 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:25.193443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.193397 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" podUID="28440a84-4173-4977-9bcd-fd0739666992" containerName="payload-processing" containerID="cri-o://0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85" gracePeriod=25 Nov 02 14:35:25.305637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.305542 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:35:25.491085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.491047 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:35:25.657244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.657192 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-routing-overlay-volume\") pod \"28440a84-4173-4977-9bcd-fd0739666992\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " Nov 02 14:35:25.657471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.657287 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-provider-credentials\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-provider-credentials\") pod \"28440a84-4173-4977-9bcd-fd0739666992\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " Nov 02 14:35:25.657558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.657519 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-plugins-config-volume\") pod \"28440a84-4173-4977-9bcd-fd0739666992\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " Nov 02 14:35:25.657726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.657632 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-kube-api-access-cqn8h\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-kube-api-access-cqn8h\") pod \"28440a84-4173-4977-9bcd-fd0739666992\" (UID: \"28440a84-4173-4977-9bcd-fd0739666992\") " Nov 02 14:35:25.657887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.657751 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-routing-overlay-volume" pod "28440a84-4173-4977-9bcd-fd0739666992" (UID: "28440a84-4173-4977-9bcd-fd0739666992"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:25.658470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.658439 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-plugins-config-volume" pod "28440a84-4173-4977-9bcd-fd0739666992" (UID: "28440a84-4173-4977-9bcd-fd0739666992"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:25.658703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.658675 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:25.658703 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.658700 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28440a84-4173-4977-9bcd-fd0739666992-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:25.660426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.660396 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-kube-api-access-cqn8h" pod "28440a84-4173-4977-9bcd-fd0739666992" (UID: "28440a84-4173-4977-9bcd-fd0739666992"). InnerVolumeSpecName "kube-api-access-cqn8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:25.660426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.660413 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-provider-credentials" pod "28440a84-4173-4977-9bcd-fd0739666992" (UID: "28440a84-4173-4977-9bcd-fd0739666992"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:25.760692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.760655 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqn8h\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-kube-api-access-cqn8h\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:25.760692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:25.760688 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28440a84-4173-4977-9bcd-fd0739666992-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:26.040494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.040398 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:26.224138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.224102 5591 generic.go:356] "Generic (PLEG): container finished" podID="28440a84-4173-4977-9bcd-fd0739666992" containerID="0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85" exitCode=0 Nov 02 14:35:26.224602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.224150 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" event={"ID":"28440a84-4173-4977-9bcd-fd0739666992","Type":"ContainerDied","Data":"0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85"} Nov 02 14:35:26.224602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.224154 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" Nov 02 14:35:26.224602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.224192 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk" event={"ID":"28440a84-4173-4977-9bcd-fd0739666992","Type":"ContainerDied","Data":"d3c637c85cd2254c4e37e4b6e19ef96bf41072e2ddadfd685fd42d2302a03b52"} Nov 02 14:35:26.224602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.224213 5591 scope.go:122] "RemoveContainer" containerID="0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85" Nov 02 14:35:26.237949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.237923 5591 scope.go:122] "RemoveContainer" containerID="0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85" Nov 02 14:35:26.238273 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:35:26.238250 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85\": container with ID starting with 0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85 not found: ID does not exist" containerID="0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85" Nov 02 14:35:26.238335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.238283 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85"} err="failed to get container status \"0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85\": rpc error: code = NotFound desc = could not find container \"0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85\": container with ID starting with 0e7771751d74ca66f1d892eed682905e9409ed8700a1110f9c9ddc9e55ee2b85 not found: ID does not exist" Nov 02 14:35:26.285974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.285937 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk"] Nov 02 14:35:26.297999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.297917 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884sdqxk"] Nov 02 14:35:26.317595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.317561 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:26.891028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:26.887961 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:27.432738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:27.432706 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28440a84-4173-4977-9bcd-fd0739666992" path="/var/lib/kubelet/pods/28440a84-4173-4977-9bcd-fd0739666992/volumes" Nov 02 14:35:28.839710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:28.839661 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:30.462145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:30.461928 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:30.804866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:30.804790 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:31.075174 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:35:31.075093 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:35:32.156470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:32.156399 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:33.201920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:33.201892 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:35:33.892945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:33.892903 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:35.481729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:35.481696 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:37.082317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:37.080891 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:38.804748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:38.804710 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:40.120195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:40.120153 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:40.310950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:40.310874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:40.982619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:40.981706 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:42.519528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.519495 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw"] Nov 02 14:35:42.520389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.520361 5591 cpu_manager.go:414] "containerMap: removing container" podUID="28440a84-4173-4977-9bcd-fd0739666992" containerName="payload-processing" Nov 02 14:35:42.520389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.520384 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28440a84-4173-4977-9bcd-fd0739666992" containerName="payload-processing" Nov 02 14:35:42.520630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.520497 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="28440a84-4173-4977-9bcd-fd0739666992" containerName="payload-processing" Nov 02 14:35:42.524136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.524118 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.543296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.543268 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw"] Nov 02 14:35:42.552258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.552227 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"agc-cred-ee3b589c119b\"" Nov 02 14:35:42.603724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.603686 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9rw\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-kube-api-access-5f9rw\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.603931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.603769 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.603931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.603919 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.604145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.604117 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.705667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.705631 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.705869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.705724 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9rw\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-kube-api-access-5f9rw\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.705869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.705775 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.706041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.705869 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.706498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.706464 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.706631 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.706564 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.708464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.708440 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.738726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.738699 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9rw\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-kube-api-access-5f9rw\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.844559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.844295 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:42.881871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:42.881575 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:43.209742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:43.209713 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw"] Nov 02 14:35:43.210575 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:35:43.210551 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b8c628_cb21_4188_b828_08851ae992c5.slice/crio-0e1d15239775869a6ca411ab07233337cd00419bb36135fd6f73681a7ebca65a WatchSource:0}: Error finding container 0e1d15239775869a6ca411ab07233337cd00419bb36135fd6f73681a7ebca65a: Status 404 returned error can't find the container with id 0e1d15239775869a6ca411ab07233337cd00419bb36135fd6f73681a7ebca65a Nov 02 14:35:43.452266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:43.452220 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" event={"ID":"19b8c628-cb21-4188-b828-08851ae992c5","Type":"ContainerStarted","Data":"8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11"} Nov 02 14:35:43.452469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:43.452275 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" event={"ID":"19b8c628-cb21-4188-b828-08851ae992c5","Type":"ContainerStarted","Data":"0e1d15239775869a6ca411ab07233337cd00419bb36135fd6f73681a7ebca65a"} Nov 02 14:35:43.452469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:43.452423 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:43.496516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:43.496418 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" podStartSLOduration=1.4964050819999999 podStartE2EDuration="1.496405082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:43.494410328 +0000 UTC m=+3092.220032097" watchObservedRunningTime="2026-11-02 14:35:43.496405082 +0000 UTC m=+3092.222026852" Nov 02 14:35:44.536769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:44.536730 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:46.167093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:46.163555 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:47.983951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:47.983915 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:48.259059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:48.255964 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:48.420127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:48.420090 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw"] Nov 02 14:35:48.423443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:48.423415 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:48.462895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:48.462864 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:49.929005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:49.928645 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:51.581110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:51.580997 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:53.280349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.280311 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:53.421057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.421016 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" podUID="19b8c628-cb21-4188-b828-08851ae992c5" containerName="payload-processing" containerID="cri-o://8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11" gracePeriod=25 Nov 02 14:35:53.701849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.701821 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:53.817172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.817088 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-kube-api-access-5f9rw\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-kube-api-access-5f9rw\") pod \"19b8c628-cb21-4188-b828-08851ae992c5\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " Nov 02 14:35:53.817172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.817134 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-routing-overlay-volume\") pod \"19b8c628-cb21-4188-b828-08851ae992c5\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " Nov 02 14:35:53.817410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.817288 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-plugins-config-volume\") pod \"19b8c628-cb21-4188-b828-08851ae992c5\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " Nov 02 14:35:53.817410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.817332 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-provider-credentials\") pod \"19b8c628-cb21-4188-b828-08851ae992c5\" (UID: \"19b8c628-cb21-4188-b828-08851ae992c5\") " Nov 02 14:35:53.817581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.817551 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-routing-overlay-volume" pod "19b8c628-cb21-4188-b828-08851ae992c5" (UID: "19b8c628-cb21-4188-b828-08851ae992c5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:53.817675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.817659 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-plugins-config-volume" pod "19b8c628-cb21-4188-b828-08851ae992c5" (UID: "19b8c628-cb21-4188-b828-08851ae992c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:53.818090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.818069 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:53.818151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.818091 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19b8c628-cb21-4188-b828-08851ae992c5-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:53.819452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.819418 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-kube-api-access-5f9rw" pod "19b8c628-cb21-4188-b828-08851ae992c5" (UID: "19b8c628-cb21-4188-b828-08851ae992c5"). InnerVolumeSpecName "kube-api-access-5f9rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:53.819452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.819447 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-provider-credentials" pod "19b8c628-cb21-4188-b828-08851ae992c5" (UID: "19b8c628-cb21-4188-b828-08851ae992c5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:53.919427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.919385 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:53.919427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:53.919425 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9rw\" (UniqueName: \"kubernetes.io/projected/19b8c628-cb21-4188-b828-08851ae992c5-kube-api-access-5f9rw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:35:54.545849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.545808 5591 generic.go:356] "Generic (PLEG): container finished" podID="19b8c628-cb21-4188-b828-08851ae992c5" containerID="8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11" exitCode=0 Nov 02 14:35:54.546350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.545871 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" Nov 02 14:35:54.546350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.545897 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" event={"ID":"19b8c628-cb21-4188-b828-08851ae992c5","Type":"ContainerDied","Data":"8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11"} Nov 02 14:35:54.546350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.545951 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw" event={"ID":"19b8c628-cb21-4188-b828-08851ae992c5","Type":"ContainerDied","Data":"0e1d15239775869a6ca411ab07233337cd00419bb36135fd6f73681a7ebca65a"} Nov 02 14:35:54.546350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.545977 5591 scope.go:122] "RemoveContainer" containerID="8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11" Nov 02 14:35:54.558465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.558438 5591 scope.go:122] "RemoveContainer" containerID="8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11" Nov 02 14:35:54.558777 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:35:54.558751 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11\": container with ID starting with 8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11 not found: ID does not exist" containerID="8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11" Nov 02 14:35:54.558843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.558791 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11"} err="failed to get container status \"8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11\": rpc error: code = NotFound desc = could not find container \"8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11\": container with ID starting with 8f5dbd74272df337612444e4d90ebe88413bdbdbbf983cb70387665a64af2e11 not found: ID does not exist" Nov 02 14:35:54.605222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.605189 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw"] Nov 02 14:35:54.618132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.618102 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-84bfntmcw"] Nov 02 14:35:54.981289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:54.981256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:55.432975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:55.432936 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b8c628-cb21-4188-b828-08851ae992c5" path="/var/lib/kubelet/pods/19b8c628-cb21-4188-b828-08851ae992c5/volumes" Nov 02 14:35:56.663812 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:56.663764 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:35:58.539916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:35:58.539875 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:00.395363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:00.395316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:01.074176 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:36:01.074137 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:36:01.990700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:01.984688 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:03.680015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:03.678012 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:05.313623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:05.313589 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:05.494439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:05.494402 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:07.264942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:07.264848 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:09.038580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:09.038540 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:10.865706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:10.865652 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:11.551190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:11.551143 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn"] Nov 02 14:36:11.679369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:11.679334 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:11.840398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:11.840313 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:12.324322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:12.324288 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:12.486403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:12.485638 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn"] Nov 02 14:36:12.520301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:12.518967 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:12.687660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:12.687569 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:12.947060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:12.946597 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:13.113214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.110720 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:13.378701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.378622 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq"] Nov 02 14:36:13.379977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.379945 5591 cpu_manager.go:414] "containerMap: removing container" podUID="19b8c628-cb21-4188-b828-08851ae992c5" containerName="payload-processing" Nov 02 14:36:13.380156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.379975 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b8c628-cb21-4188-b828-08851ae992c5" containerName="payload-processing" Nov 02 14:36:13.380226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.380200 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b8c628-cb21-4188-b828-08851ae992c5" containerName="payload-processing" Nov 02 14:36:13.410391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.410358 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.451433 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.441909 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d39fa6-openshift-default-dockercfg-8rt29\"" Nov 02 14:36:13.451433 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.448397 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq"] Nov 02 14:36:13.559030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.557744 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6950f9-9019-4d10-9cea-48841b9a16d5-istiod-ca-cert\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.559030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.557923 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-podinfo\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.559030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.557961 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-envoy\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.585010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.575127 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-token\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.585010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.575181 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-certs\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.585010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.575212 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bj7w\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-kube-api-access-5bj7w\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.585010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.575518 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.585010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.575599 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-data\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.585010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.575654 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-credential-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.682067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.676952 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-podinfo\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.701139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.701094 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-envoy\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.701446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.701422 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-token\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.701585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.701569 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-certs\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.701700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.701686 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bj7w\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-kube-api-access-5bj7w\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.702026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.702003 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.731084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.707276 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-certs\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.731084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.708420 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:13.731084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.725437 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.755006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.742213 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-data\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.755006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.742294 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-credential-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.755006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.742419 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6950f9-9019-4d10-9cea-48841b9a16d5-istiod-ca-cert\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.755006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.743401 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6950f9-9019-4d10-9cea-48841b9a16d5-istiod-ca-cert\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.755006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.743676 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-data\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.755006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.743901 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-credential-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.768005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.761810 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-envoy\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.768005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.762512 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-podinfo\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.768887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.768396 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bj7w\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-kube-api-access-5bj7w\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.769583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.769556 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-token\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:13.908011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:13.907370 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:14.036512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.036463 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:14.383753 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:36:14.383110 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6950f9_9019_4d10_9cea_48841b9a16d5.slice/crio-2ff75bc611014309357beba7fc30515a9b9ff7dd1036a4859f5f55d4e388a4b7 WatchSource:0}: Error finding container 2ff75bc611014309357beba7fc30515a9b9ff7dd1036a4859f5f55d4e388a4b7: Status 404 returned error can't find the container with id 2ff75bc611014309357beba7fc30515a9b9ff7dd1036a4859f5f55d4e388a4b7 Nov 02 14:36:14.384945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.384921 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq"] Nov 02 14:36:14.386530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.386498 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:36:14.386660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.386564 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:36:14.386660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.386599 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:36:14.474199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.474167 5591 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:36:14.474282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.474244 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:36:14.725009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.724814 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:14.782994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.782948 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" event={"ID":"4d6950f9-9019-4d10-9cea-48841b9a16d5","Type":"ContainerStarted","Data":"705fcb93444b58e530c4e16fe7c8cbde6cf2437829758b6470914fde0f6d67bb"} Nov 02 14:36:14.783196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.783035 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" event={"ID":"4d6950f9-9019-4d10-9cea-48841b9a16d5","Type":"ContainerStarted","Data":"2ff75bc611014309357beba7fc30515a9b9ff7dd1036a4859f5f55d4e388a4b7"} Nov 02 14:36:14.843507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:14.843452 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podStartSLOduration=1.843438132 podStartE2EDuration="1.843438132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:14.841603235 +0000 UTC m=+3123.567225007" watchObservedRunningTime="2026-11-02 14:36:14.843438132 +0000 UTC m=+3123.569059901" Nov 02 14:36:15.039930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:15.036947 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:15.044429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:15.044397 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:15.792452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:15.792422 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:15.793537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:15.793511 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:36:16.555819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.555609 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" podUID="d7493d54-d93f-4606-8c07-151d29accc7b" containerName="payload-processing" containerID="cri-o://57ba70da5a3c7c5f2fb5dbfddbd5cafd4303a61455956a0406b943794875b553" gracePeriod=25 Nov 02 14:36:16.579503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.579466 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:16.806666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.806577 5591 generic.go:356] "Generic (PLEG): container finished" podID="d7493d54-d93f-4606-8c07-151d29accc7b" containerID="57ba70da5a3c7c5f2fb5dbfddbd5cafd4303a61455956a0406b943794875b553" exitCode=0 Nov 02 14:36:16.807491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.806836 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" event={"ID":"d7493d54-d93f-4606-8c07-151d29accc7b","Type":"ContainerDied","Data":"57ba70da5a3c7c5f2fb5dbfddbd5cafd4303a61455956a0406b943794875b553"} Nov 02 14:36:16.810243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.810216 5591 generic.go:356] "Generic (PLEG): container finished" podID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerID="75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5" exitCode=0 Nov 02 14:36:16.810907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.810824 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" event={"ID":"0d71f880-d4ea-4d6d-9505-bccb5c82253b","Type":"ContainerDied","Data":"75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5"} Nov 02 14:36:16.811310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.811290 5591 scope.go:122] "RemoveContainer" containerID="75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5" Nov 02 14:36:16.813829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.813798 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:36:16.813943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.813878 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:36:16.813943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.813930 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:36:16.892049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.892014 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:36:16.926264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.926229 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-plugins-config-volume\") pod \"d7493d54-d93f-4606-8c07-151d29accc7b\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " Nov 02 14:36:16.926436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.926386 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-kube-api-access-ph854\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-kube-api-access-ph854\") pod \"d7493d54-d93f-4606-8c07-151d29accc7b\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " Nov 02 14:36:16.926515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.926475 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-routing-overlay-volume\") pod \"d7493d54-d93f-4606-8c07-151d29accc7b\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " Nov 02 14:36:16.926738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.926720 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-provider-credentials\") pod \"d7493d54-d93f-4606-8c07-151d29accc7b\" (UID: \"d7493d54-d93f-4606-8c07-151d29accc7b\") " Nov 02 14:36:16.928029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.927995 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-plugins-config-volume" pod "d7493d54-d93f-4606-8c07-151d29accc7b" (UID: "d7493d54-d93f-4606-8c07-151d29accc7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:16.928563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.928541 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-routing-overlay-volume" pod "d7493d54-d93f-4606-8c07-151d29accc7b" (UID: "d7493d54-d93f-4606-8c07-151d29accc7b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:16.931111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.931084 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-kube-api-access-ph854" pod "d7493d54-d93f-4606-8c07-151d29accc7b" (UID: "d7493d54-d93f-4606-8c07-151d29accc7b"). InnerVolumeSpecName "kube-api-access-ph854". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:16.931411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:16.931380 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-provider-credentials" pod "d7493d54-d93f-4606-8c07-151d29accc7b" (UID: "d7493d54-d93f-4606-8c07-151d29accc7b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:17.029402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.029365 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph854\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-kube-api-access-ph854\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:36:17.029402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.029403 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:36:17.029634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.029416 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d7493d54-d93f-4606-8c07-151d29accc7b-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:36:17.029634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.029427 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7493d54-d93f-4606-8c07-151d29accc7b-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:36:17.399871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.399836 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:17.487679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.487623 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" podUID="903850ca-63e6-4183-afb0-c1562dc22fbe" containerName="payload-processing" containerID="cri-o://20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32" gracePeriod=25 Nov 02 14:36:17.763005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.762536 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:17.801268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.801242 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:36:17.819369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.819342 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" Nov 02 14:36:17.819839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.819380 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn" event={"ID":"d7493d54-d93f-4606-8c07-151d29accc7b","Type":"ContainerDied","Data":"26c0447cc667f70c71d75f66e3133ca516334941d04c3068ac9e5c8f42ebd450"} Nov 02 14:36:17.819839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.819452 5591 scope.go:122] "RemoveContainer" containerID="57ba70da5a3c7c5f2fb5dbfddbd5cafd4303a61455956a0406b943794875b553" Nov 02 14:36:17.821154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.821127 5591 generic.go:356] "Generic (PLEG): container finished" podID="903850ca-63e6-4183-afb0-c1562dc22fbe" containerID="20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32" exitCode=0 Nov 02 14:36:17.821298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.821175 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" Nov 02 14:36:17.821298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.821221 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" event={"ID":"903850ca-63e6-4183-afb0-c1562dc22fbe","Type":"ContainerDied","Data":"20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32"} Nov 02 14:36:17.821298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.821286 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn" event={"ID":"903850ca-63e6-4183-afb0-c1562dc22fbe","Type":"ContainerDied","Data":"a11bd8e1aeb8fd6a95d62388f40203509992a46b4599f075862f6ea2f5433ac1"} Nov 02 14:36:17.823793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.823765 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" event={"ID":"0d71f880-d4ea-4d6d-9505-bccb5c82253b","Type":"ContainerStarted","Data":"c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851"} Nov 02 14:36:17.830196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.830178 5591 scope.go:122] "RemoveContainer" containerID="20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32" Nov 02 14:36:17.858300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.855676 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-plugins-config-volume\") pod \"903850ca-63e6-4183-afb0-c1562dc22fbe\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " Nov 02 14:36:17.858300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.855743 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-routing-overlay-volume\") pod \"903850ca-63e6-4183-afb0-c1562dc22fbe\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " Nov 02 14:36:17.858300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.855789 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-provider-credentials\") pod \"903850ca-63e6-4183-afb0-c1562dc22fbe\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " Nov 02 14:36:17.858300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.856286 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-kube-api-access-6k4cl\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-kube-api-access-6k4cl\") pod \"903850ca-63e6-4183-afb0-c1562dc22fbe\" (UID: \"903850ca-63e6-4183-afb0-c1562dc22fbe\") " Nov 02 14:36:17.858300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.856902 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-routing-overlay-volume" pod "903850ca-63e6-4183-afb0-c1562dc22fbe" (UID: "903850ca-63e6-4183-afb0-c1562dc22fbe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:17.858300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.857966 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-plugins-config-volume" pod "903850ca-63e6-4183-afb0-c1562dc22fbe" (UID: "903850ca-63e6-4183-afb0-c1562dc22fbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:17.859448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.859412 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-kube-api-access-6k4cl" pod "903850ca-63e6-4183-afb0-c1562dc22fbe" (UID: "903850ca-63e6-4183-afb0-c1562dc22fbe"). InnerVolumeSpecName "kube-api-access-6k4cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:17.860571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.860542 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-provider-credentials" pod "903850ca-63e6-4183-afb0-c1562dc22fbe" (UID: "903850ca-63e6-4183-afb0-c1562dc22fbe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:17.887544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.887512 5591 scope.go:122] "RemoveContainer" containerID="20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32" Nov 02 14:36:17.887915 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:36:17.887885 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32\": container with ID starting with 20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32 not found: ID does not exist" containerID="20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32" Nov 02 14:36:17.888023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.887928 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32"} err="failed to get container status \"20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32\": rpc error: code = NotFound desc = could not find container \"20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32\": container with ID starting with 20c29cb0c9923d874613830a73cbde843cdedd78da7632a28f5fbf6f7e595c32 not found: ID does not exist" Nov 02 14:36:17.924049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.924009 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn"] Nov 02 14:36:17.958545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.958454 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k4cl\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-kube-api-access-6k4cl\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:36:17.958545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.958492 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:36:17.958545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.958511 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903850ca-63e6-4183-afb0-c1562dc22fbe-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:36:17.958545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.958525 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903850ca-63e6-4183-afb0-c1562dc22fbe-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:36:17.961902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:17.961878 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-76cdxgfkn"] Nov 02 14:36:18.191728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:18.191692 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn"] Nov 02 14:36:18.198966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:18.198944 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:36:18.199182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:18.199163 5591 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 02 14:36:18.199226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:18.199208 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 02 14:36:18.211343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:18.211274 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-595bhbwjn"] Nov 02 14:36:18.879832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:18.879801 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:19.199397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.199344 5591 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 02 14:36:19.199593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.199425 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 02 14:36:19.400930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.400893 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:19.460136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.458962 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903850ca-63e6-4183-afb0-c1562dc22fbe" path="/var/lib/kubelet/pods/903850ca-63e6-4183-afb0-c1562dc22fbe/volumes" Nov 02 14:36:19.460866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.460731 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7493d54-d93f-4606-8c07-151d29accc7b" path="/var/lib/kubelet/pods/d7493d54-d93f-4606-8c07-151d29accc7b/volumes" Nov 02 14:36:19.531021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.530943 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2"] Nov 02 14:36:19.556013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.552228 5591 cpu_manager.go:414] "containerMap: removing container" podUID="903850ca-63e6-4183-afb0-c1562dc22fbe" containerName="payload-processing" Nov 02 14:36:19.556013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.552263 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903850ca-63e6-4183-afb0-c1562dc22fbe" containerName="payload-processing" Nov 02 14:36:19.556013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.552299 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d7493d54-d93f-4606-8c07-151d29accc7b" containerName="payload-processing" Nov 02 14:36:19.556013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.552308 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7493d54-d93f-4606-8c07-151d29accc7b" containerName="payload-processing" Nov 02 14:36:19.556013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.552554 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7493d54-d93f-4606-8c07-151d29accc7b" containerName="payload-processing" Nov 02 14:36:19.556013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.552585 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="903850ca-63e6-4183-afb0-c1562dc22fbe" containerName="payload-processing" Nov 02 14:36:19.560055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.559446 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:19.581042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.566078 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2"] Nov 02 14:36:19.653390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.640559 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d39fa6-dockercfg-kpjvk\"" Nov 02 14:36:19.653390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.640677 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d39fa6\"" Nov 02 14:36:19.705012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.673672 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:19.705012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.701088 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m"] Nov 02 14:36:19.724055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.709403 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxhv\" (UniqueName: \"kubernetes.io/projected/64bb2eb4-559d-4b7c-8833-a76ba98da86e-kube-api-access-jcxhv\") pod \"payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2\" (UID: \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:19.724055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.709718 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bb2eb4-559d-4b7c-8833-a76ba98da86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2\" (UID: \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:19.767379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.767330 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:19.838013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.829072 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxhv\" (UniqueName: \"kubernetes.io/projected/64bb2eb4-559d-4b7c-8833-a76ba98da86e-kube-api-access-jcxhv\") pod \"payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2\" (UID: \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:19.910767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.899277 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bb2eb4-559d-4b7c-8833-a76ba98da86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2\" (UID: \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:19.910767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.900327 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bb2eb4-559d-4b7c-8833-a76ba98da86e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2\" (UID: \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:19.910767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.901043 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m"] Nov 02 14:36:19.982063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:19.980948 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxhv\" (UniqueName: \"kubernetes.io/projected/64bb2eb4-559d-4b7c-8833-a76ba98da86e-kube-api-access-jcxhv\") pod \"payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2\" (UID: \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:20.002965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.002925 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e490c6-5154-4333-be08-3bf46465f4de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m\" (UID: \"54e490c6-5154-4333-be08-3bf46465f4de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:20.003243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.003206 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7q2\" (UniqueName: \"kubernetes.io/projected/54e490c6-5154-4333-be08-3bf46465f4de-kube-api-access-4g7q2\") pod \"payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m\" (UID: \"54e490c6-5154-4333-be08-3bf46465f4de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:20.104537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.104503 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7q2\" (UniqueName: \"kubernetes.io/projected/54e490c6-5154-4333-be08-3bf46465f4de-kube-api-access-4g7q2\") pod \"payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m\" (UID: \"54e490c6-5154-4333-be08-3bf46465f4de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:20.104716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.104659 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e490c6-5154-4333-be08-3bf46465f4de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m\" (UID: \"54e490c6-5154-4333-be08-3bf46465f4de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:20.105367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.105345 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e490c6-5154-4333-be08-3bf46465f4de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m\" (UID: \"54e490c6-5154-4333-be08-3bf46465f4de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:20.129678 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.129646 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7q2\" (UniqueName: \"kubernetes.io/projected/54e490c6-5154-4333-be08-3bf46465f4de-kube-api-access-4g7q2\") pod \"payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m\" (UID: \"54e490c6-5154-4333-be08-3bf46465f4de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:20.198339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.198304 5591 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 02 14:36:20.198535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.198391 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 02 14:36:20.256059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.254533 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:20.314415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.313301 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:20.571945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.571911 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2"] Nov 02 14:36:20.572099 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:36:20.572029 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bb2eb4_559d_4b7c_8833_a76ba98da86e.slice/crio-26e671963ee93bb16c0dbbf091d4a1f4eff28c25b87067ee7982fbdc5bee043b WatchSource:0}: Error finding container 26e671963ee93bb16c0dbbf091d4a1f4eff28c25b87067ee7982fbdc5bee043b: Status 404 returned error can't find the container with id 26e671963ee93bb16c0dbbf091d4a1f4eff28c25b87067ee7982fbdc5bee043b Nov 02 14:36:20.715362 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:36:20.715334 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e490c6_5154_4333_be08_3bf46465f4de.slice/crio-94946849de04c724a790fc09af6d628ac4d4835820a4568032fa196b347329f3 WatchSource:0}: Error finding container 94946849de04c724a790fc09af6d628ac4d4835820a4568032fa196b347329f3: Status 404 returned error can't find the container with id 94946849de04c724a790fc09af6d628ac4d4835820a4568032fa196b347329f3 Nov 02 14:36:20.716453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.716427 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m"] Nov 02 14:36:20.898092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.898035 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" event={"ID":"64bb2eb4-559d-4b7c-8833-a76ba98da86e","Type":"ContainerStarted","Data":"8cbd1d7285fe5fc4f0d985fe3e05d44bcb422e0a3e9ac9698fdd4957821aeb7f"} Nov 02 14:36:20.898660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.898117 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:20.898660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.898141 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" event={"ID":"64bb2eb4-559d-4b7c-8833-a76ba98da86e","Type":"ContainerStarted","Data":"26e671963ee93bb16c0dbbf091d4a1f4eff28c25b87067ee7982fbdc5bee043b"} Nov 02 14:36:20.899840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.899814 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" event={"ID":"54e490c6-5154-4333-be08-3bf46465f4de","Type":"ContainerStarted","Data":"8c9eb39407805985674b206cabf468e95c94726509bcccca8486fa66844d7499"} Nov 02 14:36:20.899970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.899862 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" event={"ID":"54e490c6-5154-4333-be08-3bf46465f4de","Type":"ContainerStarted","Data":"94946849de04c724a790fc09af6d628ac4d4835820a4568032fa196b347329f3"} Nov 02 14:36:20.899970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:20.899893 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:21.041721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:21.041653 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" podStartSLOduration=2.04163242 podStartE2EDuration="2.04163242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:20.961288965 +0000 UTC m=+3129.686910736" watchObservedRunningTime="2026-11-02 14:36:21.04163242 +0000 UTC m=+3129.767254190" Nov 02 14:36:21.042521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:21.042471 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" podStartSLOduration=2.042455716 podStartE2EDuration="2.042455716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:21.040122359 +0000 UTC m=+3129.765744132" watchObservedRunningTime="2026-11-02 14:36:21.042455716 +0000 UTC m=+3129.768077488" Nov 02 14:36:21.066206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:21.066168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:21.209168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:21.199364 5591 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 02 14:36:21.209168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:21.199437 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 02 14:36:22.199337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:22.199297 5591 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 02 14:36:22.199837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:22.199364 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 02 14:36:22.548063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:22.547632 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:22.888735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:22.888697 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:23.118664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:23.118581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:23.198845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:23.198762 5591 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Nov 02 14:36:23.198845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:23.198819 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Nov 02 14:36:23.857339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:23.856125 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:24.161148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:24.161094 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:24.203031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:24.202962 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:36:24.203507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:24.203488 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:36:24.204280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:24.204259 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:36:25.110704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:25.110666 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:26.203542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:26.203486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:26.389574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:26.389528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:26.890370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:26.890329 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:28.682549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:28.682514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:30.386606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:30.386570 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:31.075814 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:36:31.075776 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:36:31.910907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:31.910872 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:36:31.911347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:31.911324 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:36:32.092745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:32.092705 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:33.859006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:33.856164 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:35.539099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:35.524645 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:37.499845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:37.499791 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:37.759259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:37.759171 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:37.964339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:37.964302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:39.462635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.462601 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:39.589785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.589740 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl"] Nov 02 14:36:39.594800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.594775 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:39.613822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.613792 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fe1881\"" Nov 02 14:36:39.619139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.616243 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl"] Nov 02 14:36:39.623883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.623846 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fe1881-dockercfg-dst9l\"" Nov 02 14:36:39.639669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.639631 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtcfc\" (UniqueName: \"kubernetes.io/projected/eda8bf81-a62c-460d-be49-dd42b08d7bae-kube-api-access-qtcfc\") pod \"payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl\" (UID: \"eda8bf81-a62c-460d-be49-dd42b08d7bae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:39.639864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.639839 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda8bf81-a62c-460d-be49-dd42b08d7bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl\" (UID: \"eda8bf81-a62c-460d-be49-dd42b08d7bae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:39.660090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.660048 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:39.784067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.753028 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda8bf81-a62c-460d-be49-dd42b08d7bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl\" (UID: \"eda8bf81-a62c-460d-be49-dd42b08d7bae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:39.808004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.784659 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcfc\" (UniqueName: \"kubernetes.io/projected/eda8bf81-a62c-460d-be49-dd42b08d7bae-kube-api-access-qtcfc\") pod \"payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl\" (UID: \"eda8bf81-a62c-460d-be49-dd42b08d7bae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:39.920012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.810150 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda8bf81-a62c-460d-be49-dd42b08d7bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl\" (UID: \"eda8bf81-a62c-460d-be49-dd42b08d7bae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:39.945163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:39.945129 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcfc\" (UniqueName: \"kubernetes.io/projected/eda8bf81-a62c-460d-be49-dd42b08d7bae-kube-api-access-qtcfc\") pod \"payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl\" (UID: \"eda8bf81-a62c-460d-be49-dd42b08d7bae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:40.210143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:40.210108 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:40.527164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:40.527133 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl"] Nov 02 14:36:40.527599 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:36:40.527565 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda8bf81_a62c_460d_be49_dd42b08d7bae.slice/crio-ea4a3743945cb3eb956030a1e822e198cd6145e0e84c0f022d77d809a1f19546 WatchSource:0}: Error finding container ea4a3743945cb3eb956030a1e822e198cd6145e0e84c0f022d77d809a1f19546: Status 404 returned error can't find the container with id ea4a3743945cb3eb956030a1e822e198cd6145e0e84c0f022d77d809a1f19546 Nov 02 14:36:41.055940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:41.055899 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" event={"ID":"eda8bf81-a62c-460d-be49-dd42b08d7bae","Type":"ContainerStarted","Data":"189f3d6b3710ffaa8c5f4f370223b012d67955e61c88971874f065cf7ba14158"} Nov 02 14:36:41.056144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:41.055957 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" event={"ID":"eda8bf81-a62c-460d-be49-dd42b08d7bae","Type":"ContainerStarted","Data":"ea4a3743945cb3eb956030a1e822e198cd6145e0e84c0f022d77d809a1f19546"} Nov 02 14:36:41.056144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:41.055996 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:41.109940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:41.109882 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" podStartSLOduration=2.109868256 podStartE2EDuration="2.109868256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:41.107404122 +0000 UTC m=+3149.833025875" watchObservedRunningTime="2026-11-02 14:36:41.109868256 +0000 UTC m=+3149.835490060" Nov 02 14:36:41.650684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:41.650433 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:42.387624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:42.387591 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:36:43.388004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:43.387166 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:45.304822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:45.300609 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:47.211354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:47.211323 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:48.884763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:48.884719 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:50.605449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:50.605411 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:52.067765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:52.067735 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:36:52.364205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:52.364110 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:54.163907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:54.163562 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:55.782547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:55.777522 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:57.873867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:57.873830 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:36:59.691393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:36:59.691356 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:00.466852 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:00.466819 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:00.918203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:00.918170 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:01.075704 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:37:01.075668 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:37:01.430009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:01.415916 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:01.840396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:01.840289 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:02.246464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:02.246431 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:03.788740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:03.788669 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:05.499637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:05.499581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:05.706483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:05.706449 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:07.248594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.248555 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:07.330896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.330858 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl"] Nov 02 14:37:07.337889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.337855 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:07.358751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.358715 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl"] Nov 02 14:37:07.365300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.365269 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtttf\" (UniqueName: \"kubernetes.io/projected/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-kube-api-access-wtttf\") pod \"payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl\" (UID: \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:07.365471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.365320 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl\" (UID: \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:07.378964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.378934 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e1136e-dockercfg-strqg\"" Nov 02 14:37:07.378964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.378936 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e1136e\"" Nov 02 14:37:07.471440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.471392 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtttf\" (UniqueName: \"kubernetes.io/projected/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-kube-api-access-wtttf\") pod \"payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl\" (UID: \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:07.485009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.471687 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl\" (UID: \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:07.485009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.472656 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl\" (UID: \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:07.569309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.569218 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:07.763586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.763556 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtttf\" (UniqueName: \"kubernetes.io/projected/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-kube-api-access-wtttf\") pod \"payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl\" (UID: \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:07.959742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.959688 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:07.989250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:07.989219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:08.272601 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:37:08.272573 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825bfd14_a3a6_4436_9286_3f3ea1f3fd49.slice/crio-e82900049c5c7a57c08c30ec7d611669595a3f7165a267bedec11884111d24fd WatchSource:0}: Error finding container e82900049c5c7a57c08c30ec7d611669595a3f7165a267bedec11884111d24fd: Status 404 returned error can't find the container with id e82900049c5c7a57c08c30ec7d611669595a3f7165a267bedec11884111d24fd Nov 02 14:37:08.273049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:08.272779 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl"] Nov 02 14:37:08.424891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:08.424846 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" event={"ID":"825bfd14-a3a6-4436-9286-3f3ea1f3fd49","Type":"ContainerStarted","Data":"cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879"} Nov 02 14:37:08.425121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:08.424911 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" event={"ID":"825bfd14-a3a6-4436-9286-3f3ea1f3fd49","Type":"ContainerStarted","Data":"e82900049c5c7a57c08c30ec7d611669595a3f7165a267bedec11884111d24fd"} Nov 02 14:37:08.425121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:08.424953 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:08.470560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:08.470510 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" podStartSLOduration=1.4704958430000001 podStartE2EDuration="1.470495843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:08.46609506 +0000 UTC m=+3177.191716832" watchObservedRunningTime="2026-11-02 14:37:08.470495843 +0000 UTC m=+3177.196117648" Nov 02 14:37:09.847829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:09.847794 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:10.040565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:10.040529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:10.836089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:10.807364 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58"] Nov 02 14:37:10.857635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:10.857602 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58"] Nov 02 14:37:10.858390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:10.858339 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:10.961921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:10.961883 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbv62\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-kube-api-access-pbv62\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:10.962154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:10.962124 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:10.962218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:10.962161 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:10.962275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:10.962256 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.066007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.065240 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.066007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.065298 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.066007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.065394 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.066007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.065596 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbv62\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-kube-api-access-pbv62\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.069007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.066839 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.069007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.067616 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.092052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.091678 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.098049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.097974 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbv62\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-kube-api-access-pbv62\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457twv58\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.189781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.188377 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:11.697305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.697268 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58"] Nov 02 14:37:11.698675 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:37:11.698643 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9dca60_7ad5_4745_b585_e02c74957a54.slice/crio-53e78361570d79b431e5df0693cc3e99de36e45be95483fa4f61f569c446e74b WatchSource:0}: Error finding container 53e78361570d79b431e5df0693cc3e99de36e45be95483fa4f61f569c446e74b: Status 404 returned error can't find the container with id 53e78361570d79b431e5df0693cc3e99de36e45be95483fa4f61f569c446e74b Nov 02 14:37:11.931040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.931005 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts"] Nov 02 14:37:11.935966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.935945 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:11.959945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:11.959910 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts"] Nov 02 14:37:12.034551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.034514 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l2j2\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-kube-api-access-2l2j2\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.034777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.034721 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.034860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.034832 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.035084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.035064 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.136225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.136180 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.136427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.136272 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.136504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.136471 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.136594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.136579 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l2j2\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-kube-api-access-2l2j2\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.136876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.136853 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.137215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.137190 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.138919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.138895 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.180023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.164343 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l2j2\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-kube-api-access-2l2j2\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.211067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.210876 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:12.254727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.254699 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:12.490498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.490410 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" event={"ID":"5e9dca60-7ad5-4745-b585-e02c74957a54","Type":"ContainerStarted","Data":"f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507"} Nov 02 14:37:12.490498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.490474 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" event={"ID":"5e9dca60-7ad5-4745-b585-e02c74957a54","Type":"ContainerStarted","Data":"53e78361570d79b431e5df0693cc3e99de36e45be95483fa4f61f569c446e74b"} Nov 02 14:37:12.490753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.490505 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:12.540508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.540455 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" podStartSLOduration=2.5404412819999997 podStartE2EDuration="2.540441282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:12.538882656 +0000 UTC m=+3181.264504425" watchObservedRunningTime="2026-11-02 14:37:12.540441282 +0000 UTC m=+3181.266063069" Nov 02 14:37:12.564180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:12.564148 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts"] Nov 02 14:37:12.564352 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:37:12.564330 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98303577_32e2_4aed_b86d_9d3d5dfadd7e.slice/crio-16c0402752b014b3ec405fd5a76bb277144034caa2b0d662b2e227883d4f849b WatchSource:0}: Error finding container 16c0402752b014b3ec405fd5a76bb277144034caa2b0d662b2e227883d4f849b: Status 404 returned error can't find the container with id 16c0402752b014b3ec405fd5a76bb277144034caa2b0d662b2e227883d4f849b Nov 02 14:37:13.499755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:13.499718 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" event={"ID":"98303577-32e2-4aed-b86d-9d3d5dfadd7e","Type":"ContainerStarted","Data":"d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046"} Nov 02 14:37:13.500225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:13.499774 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" event={"ID":"98303577-32e2-4aed-b86d-9d3d5dfadd7e","Type":"ContainerStarted","Data":"16c0402752b014b3ec405fd5a76bb277144034caa2b0d662b2e227883d4f849b"} Nov 02 14:37:13.500225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:13.499803 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:13.554367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:13.554317 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" podStartSLOduration=2.5543021489999997 podStartE2EDuration="2.554302149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:13.545113015 +0000 UTC m=+3182.270734785" watchObservedRunningTime="2026-11-02 14:37:13.554302149 +0000 UTC m=+3182.279923918" Nov 02 14:37:13.953009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:13.948217 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:15.695012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:15.692242 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:17.586119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:17.585207 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:19.336851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:19.336796 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:19.439842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:19.439812 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:37:21.097005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:21.092082 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:23.643627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:23.643585 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:23.823399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:23.823356 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq"] Nov 02 14:37:24.510682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:24.510652 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:24.594425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:24.594392 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6"] Nov 02 14:37:25.277231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:25.277191 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:25.481907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:25.481873 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:26.301609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:26.301572 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:26.641583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:26.641542 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:28.430627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:28.430588 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:28.824264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:28.824161 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" podUID="479119f4-57dc-429c-aee8-69dadd26647a" containerName="payload-processing" containerID="cri-o://83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0" gracePeriod=25 Nov 02 14:37:29.186828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.186803 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:37:29.302373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.302338 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-plugins-config-volume\") pod \"479119f4-57dc-429c-aee8-69dadd26647a\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " Nov 02 14:37:29.302561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.302499 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-kube-api-access-rgggq\") pod \"479119f4-57dc-429c-aee8-69dadd26647a\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " Nov 02 14:37:29.302630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.302615 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-routing-overlay-volume\") pod \"479119f4-57dc-429c-aee8-69dadd26647a\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " Nov 02 14:37:29.302718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.302704 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-provider-credentials\") pod \"479119f4-57dc-429c-aee8-69dadd26647a\" (UID: \"479119f4-57dc-429c-aee8-69dadd26647a\") " Nov 02 14:37:29.302768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.302744 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-plugins-config-volume" pod "479119f4-57dc-429c-aee8-69dadd26647a" (UID: "479119f4-57dc-429c-aee8-69dadd26647a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:29.303017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.302977 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-routing-overlay-volume" pod "479119f4-57dc-429c-aee8-69dadd26647a" (UID: "479119f4-57dc-429c-aee8-69dadd26647a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:29.303257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.303243 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:29.303307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.303260 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/479119f4-57dc-429c-aee8-69dadd26647a-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:29.304752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.304731 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-provider-credentials" pod "479119f4-57dc-429c-aee8-69dadd26647a" (UID: "479119f4-57dc-429c-aee8-69dadd26647a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:29.304814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.304783 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-kube-api-access-rgggq" pod "479119f4-57dc-429c-aee8-69dadd26647a" (UID: "479119f4-57dc-429c-aee8-69dadd26647a"). InnerVolumeSpecName "kube-api-access-rgggq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:29.405314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.405281 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:29.405314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.405309 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/479119f4-57dc-429c-aee8-69dadd26647a-kube-api-access-rgggq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:29.595362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.595291 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" podUID="f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" containerName="payload-processing" containerID="cri-o://3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93" gracePeriod=25 Nov 02 14:37:29.710821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.710785 5591 generic.go:356] "Generic (PLEG): container finished" podID="479119f4-57dc-429c-aee8-69dadd26647a" containerID="83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0" exitCode=0 Nov 02 14:37:29.711015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.710829 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" Nov 02 14:37:29.711015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.710861 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" event={"ID":"479119f4-57dc-429c-aee8-69dadd26647a","Type":"ContainerDied","Data":"83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0"} Nov 02 14:37:29.711015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.710910 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" event={"ID":"479119f4-57dc-429c-aee8-69dadd26647a","Type":"ContainerDied","Data":"2755a8fa2bf8ba896d6c9ba665f41bad097a2cc0f0104233ab214098e765720a"} Nov 02 14:37:29.711015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.710931 5591 scope.go:122] "RemoveContainer" containerID="83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0" Nov 02 14:37:29.721139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.721116 5591 scope.go:122] "RemoveContainer" containerID="83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0" Nov 02 14:37:29.721430 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:37:29.721409 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0\": container with ID starting with 83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0 not found: ID does not exist" containerID="83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0" Nov 02 14:37:29.721501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.721443 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0"} err="failed to get container status \"83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0\": rpc error: code = NotFound desc = could not find container \"83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0\": container with ID starting with 83a32804a076d2526bf604233e89f5aba27ecb96c57d16e1a511429806eabea0 not found: ID does not exist" Nov 02 14:37:29.773321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.773285 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq"] Nov 02 14:37:29.781325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.781291 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq"] Nov 02 14:37:29.926580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:29.926554 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:37:30.019124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.019029 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-kube-api-access-2l8nd\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-kube-api-access-2l8nd\") pod \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " Nov 02 14:37:30.019124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.019098 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-plugins-config-volume\") pod \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " Nov 02 14:37:30.019445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.019280 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-provider-credentials\") pod \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " Nov 02 14:37:30.019571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.019546 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-routing-overlay-volume\") pod \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\" (UID: \"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005\") " Nov 02 14:37:30.019727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.019605 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-plugins-config-volume" pod "f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" (UID: "f4dc64d7-c8ee-4a2f-a154-b3ce1d027005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:30.020128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.020104 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-routing-overlay-volume" pod "f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" (UID: "f4dc64d7-c8ee-4a2f-a154-b3ce1d027005"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:30.020354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.020335 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:30.020445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.020358 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:30.022145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.021781 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-kube-api-access-2l8nd" pod "f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" (UID: "f4dc64d7-c8ee-4a2f-a154-b3ce1d027005"). InnerVolumeSpecName "kube-api-access-2l8nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:30.023120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.022608 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-provider-credentials" pod "f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" (UID: "f4dc64d7-c8ee-4a2f-a154-b3ce1d027005"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:30.071085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.071041 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-68d5pw4dq" podUID="479119f4-57dc-429c-aee8-69dadd26647a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.249:50052\" within 1s: context deadline exceeded" Nov 02 14:37:30.071270 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:37:30.071187 5591 logging.go:55] [core] [Channel #9844 SubChannel #9845]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.249:50052", ServerName: "10.133.0.249:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.249:50052: operation was canceled" Nov 02 14:37:30.121829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.121795 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l8nd\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-kube-api-access-2l8nd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:30.121829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.121824 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:30.171596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.171553 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:30.719618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.719586 5591 generic.go:356] "Generic (PLEG): container finished" podID="f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" containerID="3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93" exitCode=0 Nov 02 14:37:30.720140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.719634 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" Nov 02 14:37:30.720140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.719671 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" event={"ID":"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005","Type":"ContainerDied","Data":"3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93"} Nov 02 14:37:30.720140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.719730 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6" event={"ID":"f4dc64d7-c8ee-4a2f-a154-b3ce1d027005","Type":"ContainerDied","Data":"2a4f58cee3b500740410bca839ac4fd5a5b13793588d7e986f556870b26bc32c"} Nov 02 14:37:30.720140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.719755 5591 scope.go:122] "RemoveContainer" containerID="3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93" Nov 02 14:37:30.757717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.756163 5591 scope.go:122] "RemoveContainer" containerID="3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93" Nov 02 14:37:30.757717 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:37:30.757390 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93\": container with ID starting with 3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93 not found: ID does not exist" containerID="3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93" Nov 02 14:37:30.757717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.757420 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93"} err="failed to get container status \"3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93\": rpc error: code = NotFound desc = could not find container \"3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93\": container with ID starting with 3bbc312a0f18da4d33960a2d1abb117147f0fe875e11762ed0f0333d50ceaa93 not found: ID does not exist" Nov 02 14:37:30.798402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.798373 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6"] Nov 02 14:37:30.802074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:30.802039 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-647bxvwp6"] Nov 02 14:37:31.085006 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:37:31.084903 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:37:31.458146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:31.458104 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479119f4-57dc-429c-aee8-69dadd26647a" path="/var/lib/kubelet/pods/479119f4-57dc-429c-aee8-69dadd26647a/volumes" Nov 02 14:37:31.458705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:31.458686 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" path="/var/lib/kubelet/pods/f4dc64d7-c8ee-4a2f-a154-b3ce1d027005/volumes" Nov 02 14:37:31.847235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:31.847126 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:32.019607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:32.019568 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:32.214090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:32.214059 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:33.186615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:33.186582 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58"] Nov 02 14:37:33.350233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:33.349805 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:33.567061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:33.563874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:34.032022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:34.026910 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:34.215881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:34.215843 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:34.354785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:34.354693 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts"] Nov 02 14:37:34.487791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:34.487755 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:35.922296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:35.922253 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:36.931477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:36.931442 5591 generic.go:356] "Generic (PLEG): container finished" podID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerID="705fcb93444b58e530c4e16fe7c8cbde6cf2437829758b6470914fde0f6d67bb" exitCode=0 Nov 02 14:37:36.931918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:36.931514 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" event={"ID":"4d6950f9-9019-4d10-9cea-48841b9a16d5","Type":"ContainerDied","Data":"705fcb93444b58e530c4e16fe7c8cbde6cf2437829758b6470914fde0f6d67bb"} Nov 02 14:37:36.931962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:36.931931 5591 scope.go:122] "RemoveContainer" containerID="705fcb93444b58e530c4e16fe7c8cbde6cf2437829758b6470914fde0f6d67bb" Nov 02 14:37:36.934055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:36.934027 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:37:36.934136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:36.934096 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:37:36.934136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:36.934127 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:37:37.811370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:37.811329 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:37.941189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:37.941155 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" event={"ID":"4d6950f9-9019-4d10-9cea-48841b9a16d5","Type":"ContainerStarted","Data":"05cf7483c61b086ec2f45729d6e411832966bd1b9a7ccc15d23a773ef5ea0e1d"} Nov 02 14:37:38.037355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.037319 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:37:38.037537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.037495 5591 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 14:37:38.037587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.037544 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 14:37:38.186976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.186934 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" podUID="5e9dca60-7ad5-4745-b585-e02c74957a54" containerName="payload-processing" containerID="cri-o://f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507" gracePeriod=25 Nov 02 14:37:38.515256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.515226 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:38.586373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.586344 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-routing-overlay-volume\") pod \"5e9dca60-7ad5-4745-b585-e02c74957a54\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " Nov 02 14:37:38.586547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.586390 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-kube-api-access-pbv62\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-kube-api-access-pbv62\") pod \"5e9dca60-7ad5-4745-b585-e02c74957a54\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " Nov 02 14:37:38.586547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.586441 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-plugins-config-volume\") pod \"5e9dca60-7ad5-4745-b585-e02c74957a54\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " Nov 02 14:37:38.586733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.586713 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-provider-credentials\") pod \"5e9dca60-7ad5-4745-b585-e02c74957a54\" (UID: \"5e9dca60-7ad5-4745-b585-e02c74957a54\") " Nov 02 14:37:38.586823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.586791 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-routing-overlay-volume" pod "5e9dca60-7ad5-4745-b585-e02c74957a54" (UID: "5e9dca60-7ad5-4745-b585-e02c74957a54"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:38.586951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.586929 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-plugins-config-volume" pod "5e9dca60-7ad5-4745-b585-e02c74957a54" (UID: "5e9dca60-7ad5-4745-b585-e02c74957a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:38.587524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.587491 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:38.587524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.587515 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9dca60-7ad5-4745-b585-e02c74957a54-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:38.589550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.589525 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-kube-api-access-pbv62" pod "5e9dca60-7ad5-4745-b585-e02c74957a54" (UID: "5e9dca60-7ad5-4745-b585-e02c74957a54"). InnerVolumeSpecName "kube-api-access-pbv62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:38.589660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.589560 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-provider-credentials" pod "5e9dca60-7ad5-4745-b585-e02c74957a54" (UID: "5e9dca60-7ad5-4745-b585-e02c74957a54"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:38.689613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.689575 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:38.689613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.689607 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbv62\" (UniqueName: \"kubernetes.io/projected/5e9dca60-7ad5-4745-b585-e02c74957a54-kube-api-access-pbv62\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:38.964347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.964307 5591 generic.go:356] "Generic (PLEG): container finished" podID="5e9dca60-7ad5-4745-b585-e02c74957a54" containerID="f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507" exitCode=0 Nov 02 14:37:38.964769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.964352 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" Nov 02 14:37:38.964769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.964378 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" event={"ID":"5e9dca60-7ad5-4745-b585-e02c74957a54","Type":"ContainerDied","Data":"f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507"} Nov 02 14:37:38.964769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.964433 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58" event={"ID":"5e9dca60-7ad5-4745-b585-e02c74957a54","Type":"ContainerDied","Data":"53e78361570d79b431e5df0693cc3e99de36e45be95483fa4f61f569c446e74b"} Nov 02 14:37:38.964769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.964459 5591 scope.go:122] "RemoveContainer" containerID="f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507" Nov 02 14:37:38.976093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.976077 5591 scope.go:122] "RemoveContainer" containerID="f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507" Nov 02 14:37:38.976351 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:37:38.976331 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507\": container with ID starting with f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507 not found: ID does not exist" containerID="f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507" Nov 02 14:37:38.976430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:38.976362 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507"} err="failed to get container status \"f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507\": rpc error: code = NotFound desc = could not find container \"f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507\": container with ID starting with f18492ea03982c8b8c8ff11cc2b01a1516e02359585987ff4fdbfb720ba66507 not found: ID does not exist" Nov 02 14:37:39.022963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.022933 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58"] Nov 02 14:37:39.058418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.058364 5591 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 14:37:39.058637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.058440 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 14:37:39.060377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.060347 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457twv58"] Nov 02 14:37:39.356173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.356053 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" podUID="98303577-32e2-4aed-b86d-9d3d5dfadd7e" containerName="payload-processing" containerID="cri-o://d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046" gracePeriod=25 Nov 02 14:37:39.463595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.463557 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9dca60-7ad5-4745-b585-e02c74957a54" path="/var/lib/kubelet/pods/5e9dca60-7ad5-4745-b585-e02c74957a54/volumes" Nov 02 14:37:39.618760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.611291 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:39.801676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.801649 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:39.876461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.876376 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-routing-overlay-volume\") pod \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " Nov 02 14:37:39.876652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.876539 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-provider-credentials\") pod \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " Nov 02 14:37:39.876652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.876572 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-plugins-config-volume\") pod \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " Nov 02 14:37:39.877101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.877061 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-routing-overlay-volume" pod "98303577-32e2-4aed-b86d-9d3d5dfadd7e" (UID: "98303577-32e2-4aed-b86d-9d3d5dfadd7e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:39.877101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.877075 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-kube-api-access-2l2j2\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-kube-api-access-2l2j2\") pod \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\" (UID: \"98303577-32e2-4aed-b86d-9d3d5dfadd7e\") " Nov 02 14:37:39.877723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.877691 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-plugins-config-volume" pod "98303577-32e2-4aed-b86d-9d3d5dfadd7e" (UID: "98303577-32e2-4aed-b86d-9d3d5dfadd7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:37:39.878150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.878125 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:39.878458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.878439 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98303577-32e2-4aed-b86d-9d3d5dfadd7e-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:39.879407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.879365 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-kube-api-access-2l2j2" pod "98303577-32e2-4aed-b86d-9d3d5dfadd7e" (UID: "98303577-32e2-4aed-b86d-9d3d5dfadd7e"). InnerVolumeSpecName "kube-api-access-2l2j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:39.880566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.880542 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-provider-credentials" pod "98303577-32e2-4aed-b86d-9d3d5dfadd7e" (UID: "98303577-32e2-4aed-b86d-9d3d5dfadd7e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:37:39.975145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.975100 5591 generic.go:356] "Generic (PLEG): container finished" podID="98303577-32e2-4aed-b86d-9d3d5dfadd7e" containerID="d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046" exitCode=0 Nov 02 14:37:39.975145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.975146 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" Nov 02 14:37:39.975694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.975146 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" event={"ID":"98303577-32e2-4aed-b86d-9d3d5dfadd7e","Type":"ContainerDied","Data":"d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046"} Nov 02 14:37:39.975694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.975228 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts" event={"ID":"98303577-32e2-4aed-b86d-9d3d5dfadd7e","Type":"ContainerDied","Data":"16c0402752b014b3ec405fd5a76bb277144034caa2b0d662b2e227883d4f849b"} Nov 02 14:37:39.975694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.975252 5591 scope.go:122] "RemoveContainer" containerID="d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046" Nov 02 14:37:39.980522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.980495 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:39.980522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.980521 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l2j2\" (UniqueName: \"kubernetes.io/projected/98303577-32e2-4aed-b86d-9d3d5dfadd7e-kube-api-access-2l2j2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:37:39.985838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.985820 5591 scope.go:122] "RemoveContainer" containerID="d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046" Nov 02 14:37:39.986133 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:37:39.986114 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046\": container with ID starting with d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046 not found: ID does not exist" containerID="d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046" Nov 02 14:37:39.986180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:39.986143 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046"} err="failed to get container status \"d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046\": rpc error: code = NotFound desc = could not find container \"d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046\": container with ID starting with d337fa8e6400cd3cbf610d716248a7f6df39f866060687cbc409e8fc09f6a046 not found: ID does not exist" Nov 02 14:37:40.057022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:40.053953 5591 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 14:37:40.057022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:40.054031 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 14:37:40.057657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:40.057627 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts"] Nov 02 14:37:40.060534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:40.060505 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5884hp6ts"] Nov 02 14:37:40.928364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:40.928325 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:41.056514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.056475 5591 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 14:37:41.057103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.056556 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 14:37:41.107783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.107749 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:41.297075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.294578 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:41.459977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.459918 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98303577-32e2-4aed-b86d-9d3d5dfadd7e" path="/var/lib/kubelet/pods/98303577-32e2-4aed-b86d-9d3d5dfadd7e/volumes" Nov 02 14:37:41.471570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.471532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:41.980488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.980455 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt"] Nov 02 14:37:41.981350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981327 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5e9dca60-7ad5-4745-b585-e02c74957a54" containerName="payload-processing" Nov 02 14:37:41.981350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981350 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9dca60-7ad5-4745-b585-e02c74957a54" containerName="payload-processing" Nov 02 14:37:41.981496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981402 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" containerName="payload-processing" Nov 02 14:37:41.981496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981409 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" containerName="payload-processing" Nov 02 14:37:41.981496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981424 5591 cpu_manager.go:414] "containerMap: removing container" podUID="479119f4-57dc-429c-aee8-69dadd26647a" containerName="payload-processing" Nov 02 14:37:41.981496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981430 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479119f4-57dc-429c-aee8-69dadd26647a" containerName="payload-processing" Nov 02 14:37:41.981496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981448 5591 cpu_manager.go:414] "containerMap: removing container" podUID="98303577-32e2-4aed-b86d-9d3d5dfadd7e" containerName="payload-processing" Nov 02 14:37:41.981496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981453 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98303577-32e2-4aed-b86d-9d3d5dfadd7e" containerName="payload-processing" Nov 02 14:37:41.981759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981633 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dc64d7-c8ee-4a2f-a154-b3ce1d027005" containerName="payload-processing" Nov 02 14:37:41.981759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981645 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="479119f4-57dc-429c-aee8-69dadd26647a" containerName="payload-processing" Nov 02 14:37:41.981759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981655 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="98303577-32e2-4aed-b86d-9d3d5dfadd7e" containerName="payload-processing" Nov 02 14:37:41.981759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.981663 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9dca60-7ad5-4745-b585-e02c74957a54" containerName="payload-processing" Nov 02 14:37:41.987016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:41.986974 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.014382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.014342 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606cd572-014b-499d-8622-67bfad54d0e0-istio-podinfo\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.014549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.014492 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-envoy\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.014549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.014529 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-credential-socket\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.014663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.014630 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-istio-token\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.014716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.014692 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-certs\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.014821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.014806 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5gc\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-kube-api-access-xn5gc\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.014955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.014942 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606cd572-014b-499d-8622-67bfad54d0e0-istiod-ca-cert\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.015047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.015008 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-socket\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.015047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.015029 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-data\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.066017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.055522 5591 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 14:37:42.066017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.055586 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 14:37:42.066017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.056095 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-16c822-openshift-default-dockercfg-7qv8k\"" Nov 02 14:37:42.073943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.073903 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt"] Nov 02 14:37:42.088042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.086297 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117085 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-envoy\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117142 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-credential-socket\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117195 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-istio-token\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117231 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-certs\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117366 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5gc\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-kube-api-access-xn5gc\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117535 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606cd572-014b-499d-8622-67bfad54d0e0-istiod-ca-cert\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117601 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-socket\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117621 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-data\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117658 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606cd572-014b-499d-8622-67bfad54d0e0-istio-podinfo\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117691 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-credential-socket\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117862 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-certs\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.117973 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-data\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.118097 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-socket\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.118613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.118568 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606cd572-014b-499d-8622-67bfad54d0e0-istiod-ca-cert\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.120646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.120620 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-envoy\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.121649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.121627 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606cd572-014b-499d-8622-67bfad54d0e0-istio-podinfo\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.162020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.151729 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-istio-token\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.162020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.160740 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5gc\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-kube-api-access-xn5gc\") pod \"e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.317128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.307133 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:42.317128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.308809 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:42.858768 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:37:42.857933 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606cd572_014b_499d_8622_67bfad54d0e0.slice/crio-05d23814fb16000196f2b00c120869c670cf1e25bfeccfdc578df4d0c8603712 WatchSource:0}: Error finding container 05d23814fb16000196f2b00c120869c670cf1e25bfeccfdc578df4d0c8603712: Status 404 returned error can't find the container with id 05d23814fb16000196f2b00c120869c670cf1e25bfeccfdc578df4d0c8603712 Nov 02 14:37:42.858768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.858376 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt"] Nov 02 14:37:42.860911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.860876 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:37:42.861020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.860946 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:37:42.861020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:42.860976 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:37:43.015775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.015731 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" event={"ID":"606cd572-014b-499d-8622-67bfad54d0e0","Type":"ContainerStarted","Data":"7c864929490bf6c1abec5e8566fcdff7b5e2f3f0820593cbc4cdcbdbc692b2a1"} Nov 02 14:37:43.016032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.015800 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" event={"ID":"606cd572-014b-499d-8622-67bfad54d0e0","Type":"ContainerStarted","Data":"05d23814fb16000196f2b00c120869c670cf1e25bfeccfdc578df4d0c8603712"} Nov 02 14:37:43.046128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.040307 5591 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 14:37:43.046128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.040366 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 14:37:43.067853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.066403 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" podStartSLOduration=2.066390904 podStartE2EDuration="2.066390904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:43.063644679 +0000 UTC m=+3211.789266455" watchObservedRunningTime="2026-11-02 14:37:43.066390904 +0000 UTC m=+3211.792012674" Nov 02 14:37:43.164032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.163997 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:43.308634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.308504 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:43.311753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.311428 5591 patch_prober.go:29] interesting pod/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 02 14:37:43.311753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:43.311487 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 02 14:37:44.041831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:44.041797 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:37:44.042340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:44.042320 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:37:44.042945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:44.042924 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:37:44.314127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:44.314051 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:45.061039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:45.061006 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:45.076800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:45.076759 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:45.079138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:45.079109 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:37:45.691383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:45.691350 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:45.885273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:45.885237 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:47.107455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.107417 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:47.455055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.455020 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:47.571297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.571263 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5"] Nov 02 14:37:47.576195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.576172 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:47.667008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.656653 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-16c822\"" Nov 02 14:37:47.667008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.656740 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-16c822-dockercfg-xmkng\"" Nov 02 14:37:47.706074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.690675 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d843802-b544-4a9d-aaaa-046a6607e812-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5\" (UID: \"1d843802-b544-4a9d-aaaa-046a6607e812\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:47.706074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.690747 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zj2k\" (UniqueName: \"kubernetes.io/projected/1d843802-b544-4a9d-aaaa-046a6607e812-kube-api-access-9zj2k\") pod \"payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5\" (UID: \"1d843802-b544-4a9d-aaaa-046a6607e812\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:47.716256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.715595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:47.739048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.728034 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5"] Nov 02 14:37:47.846014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.841034 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zj2k\" (UniqueName: \"kubernetes.io/projected/1d843802-b544-4a9d-aaaa-046a6607e812-kube-api-access-9zj2k\") pod \"payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5\" (UID: \"1d843802-b544-4a9d-aaaa-046a6607e812\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:47.846014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.841555 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d843802-b544-4a9d-aaaa-046a6607e812-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5\" (UID: \"1d843802-b544-4a9d-aaaa-046a6607e812\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:47.846014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.842600 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d843802-b544-4a9d-aaaa-046a6607e812-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5\" (UID: \"1d843802-b544-4a9d-aaaa-046a6607e812\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:47.927118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.913079 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k"] Nov 02 14:37:47.927118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.920443 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:47.945705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.927440 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k"] Nov 02 14:37:47.956557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.954047 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k\" (UID: \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:47.956557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.954312 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdlt\" (UniqueName: \"kubernetes.io/projected/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-kube-api-access-dvdlt\") pod \"payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k\" (UID: \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:47.956557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:47.955734 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zj2k\" (UniqueName: \"kubernetes.io/projected/1d843802-b544-4a9d-aaaa-046a6607e812-kube-api-access-9zj2k\") pod \"payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5\" (UID: \"1d843802-b544-4a9d-aaaa-046a6607e812\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:48.055518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.055486 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k\" (UID: \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:48.055706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.055643 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdlt\" (UniqueName: \"kubernetes.io/projected/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-kube-api-access-dvdlt\") pod \"payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k\" (UID: \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:48.056304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.056276 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k\" (UID: \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:48.081069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.081037 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdlt\" (UniqueName: \"kubernetes.io/projected/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-kube-api-access-dvdlt\") pod \"payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k\" (UID: \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:48.191606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.191573 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:48.292608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.292528 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:48.520993 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.520936 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5"] Nov 02 14:37:48.521481 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:37:48.521457 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d843802_b544_4a9d_aaaa_046a6607e812.slice/crio-669730f1f6e755865e2e5481191716c7604d4920d2d567a5b3af067fc13f7372 WatchSource:0}: Error finding container 669730f1f6e755865e2e5481191716c7604d4920d2d567a5b3af067fc13f7372: Status 404 returned error can't find the container with id 669730f1f6e755865e2e5481191716c7604d4920d2d567a5b3af067fc13f7372 Nov 02 14:37:48.618465 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:37:48.618434 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5939cf19_5cac_4a71_8ea7_37fd8d4c1267.slice/crio-33d8927879cdf97ead0140badb6f1709e612c998dade2dc18a9e66a445716fc3 WatchSource:0}: Error finding container 33d8927879cdf97ead0140badb6f1709e612c998dade2dc18a9e66a445716fc3: Status 404 returned error can't find the container with id 33d8927879cdf97ead0140badb6f1709e612c998dade2dc18a9e66a445716fc3 Nov 02 14:37:48.619789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.619766 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k"] Nov 02 14:37:48.919191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:48.919090 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:49.121878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:49.121836 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" event={"ID":"5939cf19-5cac-4a71-8ea7-37fd8d4c1267","Type":"ContainerStarted","Data":"93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56"} Nov 02 14:37:49.122069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:49.121910 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:37:49.122069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:49.121928 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" event={"ID":"5939cf19-5cac-4a71-8ea7-37fd8d4c1267","Type":"ContainerStarted","Data":"33d8927879cdf97ead0140badb6f1709e612c998dade2dc18a9e66a445716fc3"} Nov 02 14:37:49.123681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:49.123656 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" event={"ID":"1d843802-b544-4a9d-aaaa-046a6607e812","Type":"ContainerStarted","Data":"9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800"} Nov 02 14:37:49.123797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:49.123697 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" event={"ID":"1d843802-b544-4a9d-aaaa-046a6607e812","Type":"ContainerStarted","Data":"669730f1f6e755865e2e5481191716c7604d4920d2d567a5b3af067fc13f7372"} Nov 02 14:37:49.123797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:49.123741 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:37:49.167643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:49.167592 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" podStartSLOduration=2.1675791110000002 podStartE2EDuration="2.167579111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:49.166855775 +0000 UTC m=+3217.892477544" watchObservedRunningTime="2026-11-02 14:37:49.167579111 +0000 UTC m=+3217.893200880" Nov 02 14:37:49.211382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:49.211265 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" podStartSLOduration=2.2112501780000002 podStartE2EDuration="2.211250178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:49.208215274 +0000 UTC m=+3217.933837055" watchObservedRunningTime="2026-11-02 14:37:49.211250178 +0000 UTC m=+3217.936871995" Nov 02 14:37:50.792107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:50.792065 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:52.576016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:52.566504 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:54.296117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:54.296079 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:55.984005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:55.979350 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:57.897885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:57.897845 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:37:59.667234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:37:59.667193 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:00.134271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:00.134242 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:38:00.134449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:00.134316 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:38:01.076193 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:01.076145 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:38:01.400961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:01.400926 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:01.570457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:01.570420 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:01.845150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:01.845055 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:03.483018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:03.477593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:05.170993 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:05.170929 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:06.156022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:06.153643 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:38:06.828044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:06.827477 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:08.868896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:08.868842 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:10.189321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:10.189285 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:10.970842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:10.970798 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:12.036755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:12.036701 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:12.883321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:12.883241 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:14.542019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:14.541834 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:16.243136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:16.243104 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:17.791258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:17.791225 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:17.972212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:17.972179 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:18.074300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.074213 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv"] Nov 02 14:38:18.081162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.081125 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.109918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.109878 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.110491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.110455 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcd48\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-kube-api-access-jcd48\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.110778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.110758 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.110915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.110901 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.117282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.117245 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv"] Nov 02 14:38:18.117751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.117680 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"openshift-service-ca.crt\"" Nov 02 14:38:18.118204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.118172 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"routing-overlay\"" Nov 02 14:38:18.118348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.118284 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"agc-cred-1e3a26fb0e59\"" Nov 02 14:38:18.118412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.118359 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"payload-processing-external-model-e2e--005c2a0a-dockercfg-pqmxh\"" Nov 02 14:38:18.118412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.118376 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"kube-root-ca.crt\"" Nov 02 14:38:18.121381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.121353 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-87efd3\"" Nov 02 14:38:18.158689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.158641 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:18.212608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.212575 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcd48\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-kube-api-access-jcd48\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.212834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.212650 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.212834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.212673 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.212834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.212742 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.213384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.213352 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.213515 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.213412 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.215295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.215272 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.238100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.238064 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcd48\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-kube-api-access-jcd48\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.441848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.441813 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:18.759065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:18.751850 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:19.198538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.198501 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv"] Nov 02 14:38:19.435216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.435179 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" event={"ID":"72a0bc99-9bb2-4f2f-91f3-e0976309a67d","Type":"ContainerStarted","Data":"3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5"} Nov 02 14:38:19.435391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.435244 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" event={"ID":"72a0bc99-9bb2-4f2f-91f3-e0976309a67d","Type":"ContainerStarted","Data":"ca60db8dfabbb09ae4a4b90744c6db73af88ac2ec4e8959f4bd560f96928dd10"} Nov 02 14:38:19.435391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.435275 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:19.459537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.459458 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j"] Nov 02 14:38:19.464233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.464211 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.481798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.481744 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" podStartSLOduration=1.481727905 podStartE2EDuration="1.481727905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:19.478819307 +0000 UTC m=+3248.204441080" watchObservedRunningTime="2026-11-02 14:38:19.481727905 +0000 UTC m=+3248.207349676" Nov 02 14:38:19.497041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.490244 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j"] Nov 02 14:38:19.497041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.491613 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"routing-overlay\"" Nov 02 14:38:19.497041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.491703 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"kube-root-ca.crt\"" Nov 02 14:38:19.497041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.491866 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bc34db\"" Nov 02 14:38:19.497041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.491909 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"agc-cred-8c6ef24cb8d4\"" Nov 02 14:38:19.497041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.491922 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"openshift-service-ca.crt\"" Nov 02 14:38:19.497041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.491959 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"payload-processing-external-model-e2e--2bdbc850-dockercfg-4xlsb\"" Nov 02 14:38:19.594779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.594728 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.595046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.594783 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.595129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.595060 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.595205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.595184 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwd4j\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-kube-api-access-bwd4j\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.696740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.696701 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.696941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.696856 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwd4j\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-kube-api-access-bwd4j\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.697074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.697050 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.697246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.697085 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.699614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.698022 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.699743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.699621 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.699975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.699952 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.720903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.720837 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwd4j\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-kube-api-access-bwd4j\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:19.779605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:19.779574 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:20.066610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.066529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:20.094273 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:38:20.094242 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422068e8_06e7_4a48_8f7a_949a054a7cb8.slice/crio-acf0738759e15dc67d61e976ae60da2f2bdababefbb2f7817275caf6a8de6d3c WatchSource:0}: Error finding container acf0738759e15dc67d61e976ae60da2f2bdababefbb2f7817275caf6a8de6d3c: Status 404 returned error can't find the container with id acf0738759e15dc67d61e976ae60da2f2bdababefbb2f7817275caf6a8de6d3c Nov 02 14:38:20.094516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.094494 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j"] Nov 02 14:38:20.343128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.342166 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:20.467080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.467041 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" event={"ID":"422068e8-06e7-4a48-8f7a-949a054a7cb8","Type":"ContainerStarted","Data":"679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f"} Nov 02 14:38:20.467846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.467810 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" event={"ID":"422068e8-06e7-4a48-8f7a-949a054a7cb8","Type":"ContainerStarted","Data":"acf0738759e15dc67d61e976ae60da2f2bdababefbb2f7817275caf6a8de6d3c"} Nov 02 14:38:20.468084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.467901 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:20.514495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.514440 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" podStartSLOduration=1.514427633 podStartE2EDuration="1.514427633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:20.51173389 +0000 UTC m=+3249.237355671" watchObservedRunningTime="2026-11-02 14:38:20.514427633 +0000 UTC m=+3249.240049402" Nov 02 14:38:20.521517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.521488 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:20.640997 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:20.640921 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-1e3a26fb0e59: secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:20.640997 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:20.640966 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv: secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:20.640997 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:20.640932 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bc34db/agc-cred-8c6ef24cb8d4: secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:20.641260 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:20.641015 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j: secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:20.641260 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:20.641043 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials podName:72a0bc99-9bb2-4f2f-91f3-e0976309a67d nodeName:}" failed. No retries permitted until 2026-11-02 14:38:21.141021751 +0000 UTC m=+3249.866643499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" (UID: "72a0bc99-9bb2-4f2f-91f3-e0976309a67d") : secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:20.641260 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:20.641064 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials podName:422068e8-06e7-4a48-8f7a-949a054a7cb8 nodeName:}" failed. No retries permitted until 2026-11-02 14:38:21.141056345 +0000 UTC m=+3249.866678093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8") : secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:20.941524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.941431 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:20.956375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:20.956345 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j"] Nov 02 14:38:21.149331 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:21.149294 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bc34db/agc-cred-8c6ef24cb8d4: secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:21.149331 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:21.149325 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j: secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:21.149568 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:21.149341 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-1e3a26fb0e59: secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:21.149568 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:21.149362 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv: secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:21.149568 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:21.149399 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials podName:422068e8-06e7-4a48-8f7a-949a054a7cb8 nodeName:}" failed. No retries permitted until 2026-11-02 14:38:22.149376855 +0000 UTC m=+3250.874998604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8") : secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:21.149568 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:21.149418 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials podName:72a0bc99-9bb2-4f2f-91f3-e0976309a67d nodeName:}" failed. No retries permitted until 2026-11-02 14:38:22.149409442 +0000 UTC m=+3250.875031204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" (UID: "72a0bc99-9bb2-4f2f-91f3-e0976309a67d") : secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:21.183521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:21.183476 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:21.571049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:21.567929 5591 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" secret="" err="secret \"payload-processing-external-model-e2e--2bdbc850-dockercfg-4xlsb\" not found" Nov 02 14:38:21.660048 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:21.659977 5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-plugins-e2e-extmod-b-bc34db: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bc34db" not found Nov 02 14:38:21.660266 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:21.660092 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume podName:422068e8-06e7-4a48-8f7a-949a054a7cb8 nodeName:}" failed. No retries permitted until 2026-11-02 14:38:22.160076903 +0000 UTC m=+3250.885698650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bc34db" not found Nov 02 14:38:21.984179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:21.984130 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:22.008588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:22.008554 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv"] Nov 02 14:38:22.174321 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:22.174286 5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-plugins-e2e-extmod-b-bc34db: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bc34db" not found Nov 02 14:38:22.174515 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:22.174349 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume podName:422068e8-06e7-4a48-8f7a-949a054a7cb8 nodeName:}" failed. No retries permitted until 2026-11-02 14:38:23.17433544 +0000 UTC m=+3251.899957192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bc34db" not found Nov 02 14:38:22.174515 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:22.174401 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bc34db/agc-cred-8c6ef24cb8d4: secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:22.174515 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:22.174425 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j: secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:22.174515 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:22.174468 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials podName:422068e8-06e7-4a48-8f7a-949a054a7cb8 nodeName:}" failed. No retries permitted until 2026-11-02 14:38:24.174455908 +0000 UTC m=+3252.900077657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8") : secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:22.174515 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:22.174405 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-1e3a26fb0e59: secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:22.174515 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:22.174488 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv: secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:22.174515 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:22.174510 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials podName:72a0bc99-9bb2-4f2f-91f3-e0976309a67d nodeName:}" failed. No retries permitted until 2026-11-02 14:38:24.174503494 +0000 UTC m=+3252.900125241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" (UID: "72a0bc99-9bb2-4f2f-91f3-e0976309a67d") : secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:22.200497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:22.200469 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:23.213101 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:23.213024 5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-plugins-e2e-extmod-b-bc34db: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bc34db" not found Nov 02 14:38:23.213101 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:23.213091 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume podName:422068e8-06e7-4a48-8f7a-949a054a7cb8 nodeName:}" failed. No retries permitted until 2026-11-02 14:38:25.213076111 +0000 UTC m=+3253.938697863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bc34db" not found Nov 02 14:38:23.771629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:23.769374 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:24.233396 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:24.233362 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-1e3a26fb0e59: secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:24.233396 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:24.233391 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv: secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:24.233848 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:24.233446 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials podName:72a0bc99-9bb2-4f2f-91f3-e0976309a67d nodeName:}" failed. No retries permitted until 2026-11-02 14:38:28.233429625 +0000 UTC m=+3256.959051373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" (UID: "72a0bc99-9bb2-4f2f-91f3-e0976309a67d") : secret "agc-cred-1e3a26fb0e59" not found Nov 02 14:38:24.233848 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:24.233490 5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bc34db/agc-cred-8c6ef24cb8d4: secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:24.233848 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:24.233514 5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j: secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:24.233848 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:24.233562 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials podName:422068e8-06e7-4a48-8f7a-949a054a7cb8 nodeName:}" failed. No retries permitted until 2026-11-02 14:38:28.233549649 +0000 UTC m=+3256.959171410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8") : secret "agc-cred-8c6ef24cb8d4" not found Nov 02 14:38:25.249304 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:25.249268 5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-plugins-e2e-extmod-b-bc34db: configmap "payload-processing-external-model-plugins-e2e-extmod-b-bc34db" not found Nov 02 14:38:25.249755 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:25.249351 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume podName:422068e8-06e7-4a48-8f7a-949a054a7cb8 nodeName:}" failed. No retries permitted until 2026-11-02 14:38:29.249329954 +0000 UTC m=+3257.974951720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-bc34db" not found Nov 02 14:38:25.482062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:25.482021 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:26.480221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:26.480165 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:27.008824 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.008783 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" podUID="72a0bc99-9bb2-4f2f-91f3-e0976309a67d" containerName="payload-processing" containerID="cri-o://3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5" gracePeriod=25 Nov 02 14:38:27.245223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.244931 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:27.368207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.368177 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:27.495248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.495213 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-routing-overlay-volume\") pod \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " Nov 02 14:38:27.495650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.495387 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-plugins-config-volume\") pod \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " Nov 02 14:38:27.495650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.495449 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-kube-api-access-jcd48\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-kube-api-access-jcd48\") pod \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " Nov 02 14:38:27.495650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.495627 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials\") pod \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\" (UID: \"72a0bc99-9bb2-4f2f-91f3-e0976309a67d\") " Nov 02 14:38:27.495783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.495693 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-routing-overlay-volume" pod "72a0bc99-9bb2-4f2f-91f3-e0976309a67d" (UID: "72a0bc99-9bb2-4f2f-91f3-e0976309a67d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:27.495862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.495835 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-plugins-config-volume" pod "72a0bc99-9bb2-4f2f-91f3-e0976309a67d" (UID: "72a0bc99-9bb2-4f2f-91f3-e0976309a67d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:27.496372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.496347 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:38:27.496372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.496370 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:38:27.497671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.497645 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-kube-api-access-jcd48" pod "72a0bc99-9bb2-4f2f-91f3-e0976309a67d" (UID: "72a0bc99-9bb2-4f2f-91f3-e0976309a67d"). InnerVolumeSpecName "kube-api-access-jcd48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:27.497845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.497830 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials" pod "72a0bc99-9bb2-4f2f-91f3-e0976309a67d" (UID: "72a0bc99-9bb2-4f2f-91f3-e0976309a67d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:27.593158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.593048 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" podUID="422068e8-06e7-4a48-8f7a-949a054a7cb8" containerName="payload-processing" containerID="cri-o://679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f" gracePeriod=25 Nov 02 14:38:27.597996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.597950 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcd48\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-kube-api-access-jcd48\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:38:27.597996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.597979 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72a0bc99-9bb2-4f2f-91f3-e0976309a67d-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:38:27.628940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.628912 5591 generic.go:356] "Generic (PLEG): container finished" podID="72a0bc99-9bb2-4f2f-91f3-e0976309a67d" containerID="3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5" exitCode=0 Nov 02 14:38:27.629110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.628954 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" Nov 02 14:38:27.629110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.629003 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" event={"ID":"72a0bc99-9bb2-4f2f-91f3-e0976309a67d","Type":"ContainerDied","Data":"3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5"} Nov 02 14:38:27.629110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.629061 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv" event={"ID":"72a0bc99-9bb2-4f2f-91f3-e0976309a67d","Type":"ContainerDied","Data":"ca60db8dfabbb09ae4a4b90744c6db73af88ac2ec4e8959f4bd560f96928dd10"} Nov 02 14:38:27.629110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.629084 5591 scope.go:122] "RemoveContainer" containerID="3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5" Nov 02 14:38:27.707546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.707520 5591 scope.go:122] "RemoveContainer" containerID="3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5" Nov 02 14:38:27.708180 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:27.708146 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5\": container with ID starting with 3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5 not found: ID does not exist" containerID="3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5" Nov 02 14:38:27.708277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.708193 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5"} err="failed to get container status \"3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5\": rpc error: code = NotFound desc = could not find container \"3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5\": container with ID starting with 3ea1b0ce3ba0a56a23c07640fa08c7e57ee4134a9ff0a0a6b30573d45182d4e5 not found: ID does not exist" Nov 02 14:38:27.711917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.711896 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv"] Nov 02 14:38:27.725064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.725027 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669fk5rdv"] Nov 02 14:38:27.869411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:27.869385 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:28.004257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.004223 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials\") pod \"422068e8-06e7-4a48-8f7a-949a054a7cb8\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " Nov 02 14:38:28.004471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.004449 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-kube-api-access-bwd4j\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-kube-api-access-bwd4j\") pod \"422068e8-06e7-4a48-8f7a-949a054a7cb8\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " Nov 02 14:38:28.004545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.004477 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume\") pod \"422068e8-06e7-4a48-8f7a-949a054a7cb8\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " Nov 02 14:38:28.004545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.004506 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-routing-overlay-volume\") pod \"422068e8-06e7-4a48-8f7a-949a054a7cb8\" (UID: \"422068e8-06e7-4a48-8f7a-949a054a7cb8\") " Nov 02 14:38:28.004916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.004891 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-routing-overlay-volume" pod "422068e8-06e7-4a48-8f7a-949a054a7cb8" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:28.005044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.004914 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume" pod "422068e8-06e7-4a48-8f7a-949a054a7cb8" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:28.005215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.005195 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:38:28.005285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.005215 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/422068e8-06e7-4a48-8f7a-949a054a7cb8-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:38:28.006701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.006680 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-kube-api-access-bwd4j" pod "422068e8-06e7-4a48-8f7a-949a054a7cb8" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8"). InnerVolumeSpecName "kube-api-access-bwd4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:28.006786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.006700 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials" pod "422068e8-06e7-4a48-8f7a-949a054a7cb8" (UID: "422068e8-06e7-4a48-8f7a-949a054a7cb8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:28.106598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.106562 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwd4j\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-kube-api-access-bwd4j\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:38:28.106598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.106592 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/422068e8-06e7-4a48-8f7a-949a054a7cb8-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:38:28.640813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.640767 5591 generic.go:356] "Generic (PLEG): container finished" podID="422068e8-06e7-4a48-8f7a-949a054a7cb8" containerID="679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f" exitCode=0 Nov 02 14:38:28.640813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.640813 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" Nov 02 14:38:28.641447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.640808 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" event={"ID":"422068e8-06e7-4a48-8f7a-949a054a7cb8","Type":"ContainerDied","Data":"679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f"} Nov 02 14:38:28.641447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.640886 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j" event={"ID":"422068e8-06e7-4a48-8f7a-949a054a7cb8","Type":"ContainerDied","Data":"acf0738759e15dc67d61e976ae60da2f2bdababefbb2f7817275caf6a8de6d3c"} Nov 02 14:38:28.641447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.640915 5591 scope.go:122] "RemoveContainer" containerID="679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f" Nov 02 14:38:28.652891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.652867 5591 scope.go:122] "RemoveContainer" containerID="679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f" Nov 02 14:38:28.653179 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:28.653158 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f\": container with ID starting with 679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f not found: ID does not exist" containerID="679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f" Nov 02 14:38:28.653244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.653186 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f"} err="failed to get container status \"679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f\": rpc error: code = NotFound desc = could not find container \"679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f\": container with ID starting with 679fac795789629118d9e3315870222c1c77750b1d5fc4075bd7a433a86a030f not found: ID does not exist" Nov 02 14:38:28.704721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.704684 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j"] Nov 02 14:38:28.709630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.709601 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cdnp99j"] Nov 02 14:38:28.777217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.777181 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:28.973232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:28.973144 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:29.160202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:29.160170 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:29.367815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:29.367715 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:29.434117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:29.434085 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422068e8-06e7-4a48-8f7a-949a054a7cb8" path="/var/lib/kubelet/pods/422068e8-06e7-4a48-8f7a-949a054a7cb8/volumes" Nov 02 14:38:29.434554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:29.434540 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a0bc99-9bb2-4f2f-91f3-e0976309a67d" path="/var/lib/kubelet/pods/72a0bc99-9bb2-4f2f-91f3-e0976309a67d/volumes" Nov 02 14:38:30.033019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:30.032605 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:30.231010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:30.225663 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:31.079496 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:38:31.079462 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:38:31.265681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:31.265645 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:33.067028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:33.066065 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:34.419902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:34.419869 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:34.647756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:34.647720 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:34.855055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:34.854953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:35.755293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:35.755256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:36.183211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.183181 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:36.529372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.529289 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:36.581787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.581751 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5"] Nov 02 14:38:36.582809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.582783 5591 cpu_manager.go:414] "containerMap: removing container" podUID="72a0bc99-9bb2-4f2f-91f3-e0976309a67d" containerName="payload-processing" Nov 02 14:38:36.582809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.582807 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a0bc99-9bb2-4f2f-91f3-e0976309a67d" containerName="payload-processing" Nov 02 14:38:36.583040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.582873 5591 cpu_manager.go:414] "containerMap: removing container" podUID="422068e8-06e7-4a48-8f7a-949a054a7cb8" containerName="payload-processing" Nov 02 14:38:36.583040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.582880 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422068e8-06e7-4a48-8f7a-949a054a7cb8" containerName="payload-processing" Nov 02 14:38:36.583138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.583051 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="422068e8-06e7-4a48-8f7a-949a054a7cb8" containerName="payload-processing" Nov 02 14:38:36.583138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.583070 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a0bc99-9bb2-4f2f-91f3-e0976309a67d" containerName="payload-processing" Nov 02 14:38:36.588368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.588348 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:36.615029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.614971 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5"] Nov 02 14:38:36.623741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.623711 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c8f647-dockercfg-mxgtc\"" Nov 02 14:38:36.624191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.624127 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c8f647\"" Nov 02 14:38:36.643521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.643487 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42"] Nov 02 14:38:36.661577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.661527 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.683556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.683523 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5\" (UID: \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:36.683782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.683759 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjj7g\" (UniqueName: \"kubernetes.io/projected/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-kube-api-access-fjj7g\") pod \"payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5\" (UID: \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:36.693582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.693547 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42"] Nov 02 14:38:36.715058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.715029 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cbde78-openshift-default-dockercfg-g62p9\"" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789155 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-certs\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789230 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjj7g\" (UniqueName: \"kubernetes.io/projected/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-kube-api-access-fjj7g\") pod \"payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5\" (UID: \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789259 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-envoy\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789355 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dpx4\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-kube-api-access-2dpx4\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789420 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-credential-socket\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789482 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istiod-ca-cert\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789536 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5\" (UID: \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789607 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-token\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789786 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-podinfo\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789932 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-data\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.790059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.789996 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-socket\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.837051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.807966 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5\" (UID: \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:36.857463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.814925 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:36.857771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.857421 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjj7g\" (UniqueName: \"kubernetes.io/projected/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-kube-api-access-fjj7g\") pod \"payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5\" (UID: \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892119 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-podinfo\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892265 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-data\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892308 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-socket\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892348 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-certs\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892385 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-envoy\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892470 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpx4\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-kube-api-access-2dpx4\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892522 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-credential-socket\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892578 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istiod-ca-cert\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.966976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.892663 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-token\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:36.987034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.970543 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:36.991106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:36.978421 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-credential-socket\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.048432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.046000 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-token\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.048432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.046089 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpx4\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-kube-api-access-2dpx4\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.067146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.063939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:37.095598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.094562 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-data\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.095598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.094863 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-socket\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.095598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.095549 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istiod-ca-cert\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.096725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.096668 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-podinfo\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.117642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.117606 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-certs\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.123405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.122463 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-envoy\") pod \"e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.279605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.279560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:37.283127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.283097 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:37.447613 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:38:37.447575 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8e0f08_2dba_4a35_9c8a_07fa40182d4b.slice/crio-b5700d70ed2793d999a3e5769e6aa1b83eb3367d264e68a7f6af72f75e31414f WatchSource:0}: Error finding container b5700d70ed2793d999a3e5769e6aa1b83eb3367d264e68a7f6af72f75e31414f: Status 404 returned error can't find the container with id b5700d70ed2793d999a3e5769e6aa1b83eb3367d264e68a7f6af72f75e31414f Nov 02 14:38:37.447808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.447775 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5"] Nov 02 14:38:37.664247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.663242 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42"] Nov 02 14:38:37.664247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.664233 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:37.666778 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:38:37.666748 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd282e44_172b_4b31_81ce_ddfc8fd63a78.slice/crio-fe9e5285fc3fbaa4b704507e1aea0cdfb15e391968be3fcc2fcf147dae1fd0a6 WatchSource:0}: Error finding container fe9e5285fc3fbaa4b704507e1aea0cdfb15e391968be3fcc2fcf147dae1fd0a6: Status 404 returned error can't find the container with id fe9e5285fc3fbaa4b704507e1aea0cdfb15e391968be3fcc2fcf147dae1fd0a6 Nov 02 14:38:37.670751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.670711 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:38:37.670872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.670802 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:38:37.670872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.670844 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:38:37.724364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.724325 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" event={"ID":"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b","Type":"ContainerStarted","Data":"db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f"} Nov 02 14:38:37.724496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.724394 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" event={"ID":"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b","Type":"ContainerStarted","Data":"b5700d70ed2793d999a3e5769e6aa1b83eb3367d264e68a7f6af72f75e31414f"} Nov 02 14:38:37.724496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.724429 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:37.725968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.725943 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" event={"ID":"bd282e44-172b-4b31-81ce-ddfc8fd63a78","Type":"ContainerStarted","Data":"fe9e5285fc3fbaa4b704507e1aea0cdfb15e391968be3fcc2fcf147dae1fd0a6"} Nov 02 14:38:37.774234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:37.774173 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" podStartSLOduration=1.7741567520000001 podStartE2EDuration="1.774156752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:37.770319282 +0000 UTC m=+3266.495941053" watchObservedRunningTime="2026-11-02 14:38:37.774156752 +0000 UTC m=+3266.499778519" Nov 02 14:38:38.735486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:38.735444 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" event={"ID":"bd282e44-172b-4b31-81ce-ddfc8fd63a78","Type":"ContainerStarted","Data":"6b8fc3c7e0e74e5ee876768bb8822f005f6c476adab14f612ed9292e07cc8a7d"} Nov 02 14:38:38.798606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:38.798553 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" podStartSLOduration=2.798539027 podStartE2EDuration="2.798539027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:38.797678528 +0000 UTC m=+3267.523300299" watchObservedRunningTime="2026-11-02 14:38:38.798539027 +0000 UTC m=+3267.524160797" Nov 02 14:38:39.283554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:39.283519 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:39.289693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:39.289664 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:39.472303 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:39.472248 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:39.744979 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:39.744946 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:39.745920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:39.745901 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:38:41.287435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:41.287376 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:41.504918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:41.504833 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:43.127325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.127289 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:43.246519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.246479 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s"] Nov 02 14:38:43.251632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.251609 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:43.269848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.269804 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s"] Nov 02 14:38:43.278474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.278436 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cbde78-dockercfg-cg4pv\"" Nov 02 14:38:43.278474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.278441 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cbde78\"" Nov 02 14:38:43.315775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.315737 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:43.332750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.332652 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnb6c\" (UniqueName: \"kubernetes.io/projected/60b64700-03df-4be9-9fe6-9c37acddee29-kube-api-access-cnb6c\") pod \"payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s\" (UID: \"60b64700-03df-4be9-9fe6-9c37acddee29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:43.333100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.332864 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b64700-03df-4be9-9fe6-9c37acddee29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s\" (UID: \"60b64700-03df-4be9-9fe6-9c37acddee29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:43.446007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.445926 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnb6c\" (UniqueName: \"kubernetes.io/projected/60b64700-03df-4be9-9fe6-9c37acddee29-kube-api-access-cnb6c\") pod \"payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s\" (UID: \"60b64700-03df-4be9-9fe6-9c37acddee29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:43.448911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.446226 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b64700-03df-4be9-9fe6-9c37acddee29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s\" (UID: \"60b64700-03df-4be9-9fe6-9c37acddee29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:43.448911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.447462 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b64700-03df-4be9-9fe6-9c37acddee29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s\" (UID: \"60b64700-03df-4be9-9fe6-9c37acddee29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:43.579007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.577728 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnb6c\" (UniqueName: \"kubernetes.io/projected/60b64700-03df-4be9-9fe6-9c37acddee29-kube-api-access-cnb6c\") pod \"payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s\" (UID: \"60b64700-03df-4be9-9fe6-9c37acddee29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:43.627008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.626151 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:43.812399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.812286 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:43.868585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:43.868553 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:44.180126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:44.180078 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:44.181274 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:38:44.181247 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b64700_03df_4be9_9fe6_9c37acddee29.slice/crio-79374949838f5a32413c0291be6e9ff4d266901d253048208be8003db726f5df WatchSource:0}: Error finding container 79374949838f5a32413c0291be6e9ff4d266901d253048208be8003db726f5df: Status 404 returned error can't find the container with id 79374949838f5a32413c0291be6e9ff4d266901d253048208be8003db726f5df Nov 02 14:38:44.181396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:44.181316 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s"] Nov 02 14:38:44.819258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:44.819223 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" event={"ID":"60b64700-03df-4be9-9fe6-9c37acddee29","Type":"ContainerStarted","Data":"5f0ff6a3fa7af189a9b3fee3066db9905ae7a6df82ce9b08a4860f9c0d28ab99"} Nov 02 14:38:44.819469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:44.819278 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" event={"ID":"60b64700-03df-4be9-9fe6-9c37acddee29","Type":"ContainerStarted","Data":"79374949838f5a32413c0291be6e9ff4d266901d253048208be8003db726f5df"} Nov 02 14:38:44.819469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:44.819438 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:44.864858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:44.864801 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" podStartSLOduration=1.8647874899999999 podStartE2EDuration="1.86478749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:44.863168889 +0000 UTC m=+3273.588790659" watchObservedRunningTime="2026-11-02 14:38:44.86478749 +0000 UTC m=+3273.590409260" Nov 02 14:38:45.584585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:45.578531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:47.486005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:47.485282 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:48.738785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:48.738754 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:38:49.203649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:49.201926 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:51.291004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:51.290722 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:52.962018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:52.961624 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:54.770251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:54.769895 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:55.830181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:55.830148 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:38:56.360055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:56.357024 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:57.946571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:57.946531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:38:59.787297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:38:59.787256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:01.075433 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:39:01.075382 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:39:01.340992 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:01.340877 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:03.248220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:03.241122 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:04.993004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:04.992570 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:06.667006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:06.664744 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:08.406923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:08.406884 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:08.655765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:08.655669 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:10.046137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:10.046102 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:10.262959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:10.262740 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:11.967649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:11.967541 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:12.120033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:12.119973 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:39:12.170419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:12.170385 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:39:13.744911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:13.744851 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:15.384863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:15.384823 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:17.032689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:17.032649 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:17.430419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:17.430386 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:17.596053 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:17.596022 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:18.650700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:18.650631 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:20.487422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:20.487372 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:22.232595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:22.232556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:23.870215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:23.865563 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:25.603644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:25.603588 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:26.536933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:26.536899 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:27.197008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:27.193180 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:28.849010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:28.846214 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:29.205672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:29.205629 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:39:30.824885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:30.824846 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:31.120586 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:39:31.120486 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:39:32.657534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:32.657459 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:32.820246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:32.820213 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:33.180110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:33.180076 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:34.457820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.457777 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:34.521855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.521819 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2"] Nov 02 14:39:34.528099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.528075 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.566047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.565655 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e1136e\"/\"default-dockercfg-fbjs5\"" Nov 02 14:39:34.567900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.566320 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e1136e\"/\"openshift-service-ca.crt\"" Nov 02 14:39:34.567900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.566415 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e1136e\"/\"kube-root-ca.crt\"" Nov 02 14:39:34.567900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.566512 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e1136e\"/\"test-model-e1136e-kserve-self-signed-certs\"" Nov 02 14:39:34.568892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.568854 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2"] Nov 02 14:39:34.627443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.627408 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-dshm\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.627640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.627471 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-home\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.627640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.627627 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5l54\" (UniqueName: \"kubernetes.io/projected/56d985b1-0165-471a-aad3-5dc60b3ce843-kube-api-access-r5l54\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.627726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.627687 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d985b1-0165-471a-aad3-5dc60b3ce843-tls-certs\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.627799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.627777 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-tmp-dir\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.627840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.627824 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-model-cache\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.729788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.729699 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-dshm\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.729788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.729759 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-home\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.730049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.729808 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5l54\" (UniqueName: \"kubernetes.io/projected/56d985b1-0165-471a-aad3-5dc60b3ce843-kube-api-access-r5l54\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.730049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.729852 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d985b1-0165-471a-aad3-5dc60b3ce843-tls-certs\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.730049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.729890 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-tmp-dir\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.730049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.729920 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-model-cache\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.730319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.730288 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-home\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.730389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.730334 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-tmp-dir\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.730389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.730364 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-model-cache\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.764018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.762781 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d985b1-0165-471a-aad3-5dc60b3ce843-tls-certs\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.794008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.775908 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-dshm\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.817649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.817488 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5l54\" (UniqueName: \"kubernetes.io/projected/56d985b1-0165-471a-aad3-5dc60b3ce843-kube-api-access-r5l54\") pod \"test-model-e1136e-kserve-9854cbf77-ksrx2\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:34.836210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.836158 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:34.882389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:34.882334 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:35.357600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:35.357557 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2"] Nov 02 14:39:35.359184 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:39:35.359147 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56d985b1_0165_471a_aad3_5dc60b3ce843.slice/crio-69b9d344d9f0ea16954fa2835989fb0467875a21af8bad6a16ec723f56d48f56 WatchSource:0}: Error finding container 69b9d344d9f0ea16954fa2835989fb0467875a21af8bad6a16ec723f56d48f56: Status 404 returned error can't find the container with id 69b9d344d9f0ea16954fa2835989fb0467875a21af8bad6a16ec723f56d48f56 Nov 02 14:39:35.460868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:35.460819 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" event={"ID":"56d985b1-0165-471a-aad3-5dc60b3ce843","Type":"ContainerStarted","Data":"69b9d344d9f0ea16954fa2835989fb0467875a21af8bad6a16ec723f56d48f56"} Nov 02 14:39:36.382012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:36.381317 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:36.465047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:36.464931 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" event={"ID":"56d985b1-0165-471a-aad3-5dc60b3ce843","Type":"ContainerStarted","Data":"527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd"} Nov 02 14:39:36.514704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:36.514646 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" podStartSLOduration=2.5146304329999998 podStartE2EDuration="2.514630433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:36.509574081 +0000 UTC m=+3325.235195851" watchObservedRunningTime="2026-11-02 14:39:36.514630433 +0000 UTC m=+3325.240252239" Nov 02 14:39:36.886481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:36.886441 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:38.147635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:38.147584 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:40.862442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:40.857358 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:41.030822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:41.030708 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:41.507072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:41.507038 5591 generic.go:356] "Generic (PLEG): container finished" podID="606cd572-014b-499d-8622-67bfad54d0e0" containerID="7c864929490bf6c1abec5e8566fcdff7b5e2f3f0820593cbc4cdcbdbc692b2a1" exitCode=0 Nov 02 14:39:41.507273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:41.507117 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" event={"ID":"606cd572-014b-499d-8622-67bfad54d0e0","Type":"ContainerDied","Data":"7c864929490bf6c1abec5e8566fcdff7b5e2f3f0820593cbc4cdcbdbc692b2a1"} Nov 02 14:39:41.507712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:41.507695 5591 scope.go:122] "RemoveContainer" containerID="7c864929490bf6c1abec5e8566fcdff7b5e2f3f0820593cbc4cdcbdbc692b2a1" Nov 02 14:39:41.509843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:41.509817 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:39:41.509905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:41.509891 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:39:41.509939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:41.509924 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:39:41.661348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:41.661316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:42.516673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:42.516638 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" event={"ID":"606cd572-014b-499d-8622-67bfad54d0e0","Type":"ContainerStarted","Data":"9c5c3f770e1a620b17a353763bd22042fdc0ee13f59115f3dade7659ac7ede7f"} Nov 02 14:39:43.308051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:43.308015 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:39:43.309622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:43.309597 5591 patch_prober.go:29] interesting pod/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 02 14:39:43.309690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:43.309648 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 02 14:39:43.456172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:43.456134 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:44.308416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:44.308383 5591 patch_prober.go:29] interesting pod/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 02 14:39:44.308829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:44.308437 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 02 14:39:44.890531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:44.890484 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:45.084136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:45.080444 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:39:45.206331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:45.206122 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:45.222806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:45.222776 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:45.308309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:45.308275 5591 patch_prober.go:29] interesting pod/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 02 14:39:45.308522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:45.308349 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 02 14:39:45.584651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:45.584567 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:39:46.308049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:46.307971 5591 patch_prober.go:29] interesting pod/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 02 14:39:46.308246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:46.308076 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 02 14:39:47.124871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:47.124820 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:47.308517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:47.308474 5591 patch_prober.go:29] interesting pod/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 02 14:39:47.308725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:47.308552 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 02 14:39:48.337767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:48.337733 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:39:48.339762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:48.339734 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:39:48.651005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:48.646780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:48.820838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:48.820789 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:49.076689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.076610 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:49.436130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.436098 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w"] Nov 02 14:39:49.441585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.441560 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.466564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.466526 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w"] Nov 02 14:39:49.472600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.472571 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ea3e8f-openshift-default-dockercfg-5ngr8\"" Nov 02 14:39:49.508231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.508193 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-data\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.508412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.508238 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-credential-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.508487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.508456 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91593d10-d549-4bf5-a212-0a2afb872125-istiod-ca-cert\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.508724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.508703 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-envoy\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.508825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.508804 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-certs\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.509061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.509040 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.509140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.509124 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91593d10-d549-4bf5-a212-0a2afb872125-istio-podinfo\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.509195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.509159 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wtz\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-kube-api-access-f2wtz\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.509243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.509207 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-istio-token\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.584022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.577876 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:49.758573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.756230 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.766209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.764330 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91593d10-d549-4bf5-a212-0a2afb872125-istio-podinfo\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.789882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.789844 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wtz\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-kube-api-access-f2wtz\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.795351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.795318 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-istio-token\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.796044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.796019 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-data\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.796603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.796584 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-credential-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.796842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.796787 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91593d10-d549-4bf5-a212-0a2afb872125-istiod-ca-cert\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.799460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.799436 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-envoy\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.800806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.800782 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-certs\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.805640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.797971 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91593d10-d549-4bf5-a212-0a2afb872125-istiod-ca-cert\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.805810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.797262 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-credential-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.805939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.765775 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.808662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.808641 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-certs\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.810261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.796509 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-data\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.826322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.826289 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-envoy\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.831801 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.831770 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91593d10-d549-4bf5-a212-0a2afb872125-istio-podinfo\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.833104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.833076 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wtz\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-kube-api-access-f2wtz\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:49.837664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:49.837613 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-istio-token\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:50.064360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.064268 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:50.164575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.164157 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:50.404263 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:39:50.404232 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91593d10_d549_4bf5_a212_0a2afb872125.slice/crio-a56d91f59ff20ef75838e7fd62692c7a0f07cefa116ca458d098c8afd9fb2dfa WatchSource:0}: Error finding container a56d91f59ff20ef75838e7fd62692c7a0f07cefa116ca458d098c8afd9fb2dfa: Status 404 returned error can't find the container with id a56d91f59ff20ef75838e7fd62692c7a0f07cefa116ca458d098c8afd9fb2dfa Nov 02 14:39:50.404391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.404322 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w"] Nov 02 14:39:50.407383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.407350 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:39:50.407505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.407423 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:39:50.407505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.407488 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:39:50.490588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.490513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:50.758537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.758442 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" event={"ID":"91593d10-d549-4bf5-a212-0a2afb872125","Type":"ContainerStarted","Data":"de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75"} Nov 02 14:39:50.758537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.758505 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" event={"ID":"91593d10-d549-4bf5-a212-0a2afb872125","Type":"ContainerStarted","Data":"a56d91f59ff20ef75838e7fd62692c7a0f07cefa116ca458d098c8afd9fb2dfa"} Nov 02 14:39:50.844874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:50.844821 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" podStartSLOduration=1.8448062969999999 podStartE2EDuration="1.844806297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:50.844461793 +0000 UTC m=+3339.570083574" watchObservedRunningTime="2026-11-02 14:39:50.844806297 +0000 UTC m=+3339.570428068" Nov 02 14:39:51.065359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:51.065278 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:51.071845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:51.071814 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:51.766050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:51.766006 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:51.766877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:51.766854 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:39:52.299952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:52.286755 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:53.343202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:53.343118 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:53.519298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:53.519265 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:54.285665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:54.285627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:54.989263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:54.989226 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:55.118960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.118919 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4"] Nov 02 14:39:55.123998 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.123969 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:55.143704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.143676 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4"] Nov 02 14:39:55.151362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.151330 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ea3e8f\"" Nov 02 14:39:55.151362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.151356 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ea3e8f-dockercfg-nnx58\"" Nov 02 14:39:55.161745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.161717 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:55.208008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.207744 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rww9\" (UniqueName: \"kubernetes.io/projected/ee57b283-ffc3-4310-9113-6568a46d0034-kube-api-access-2rww9\") pod \"payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4\" (UID: \"ee57b283-ffc3-4310-9113-6568a46d0034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:55.208008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.207808 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee57b283-ffc3-4310-9113-6568a46d0034-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4\" (UID: \"ee57b283-ffc3-4310-9113-6568a46d0034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:55.338063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.314248 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rww9\" (UniqueName: \"kubernetes.io/projected/ee57b283-ffc3-4310-9113-6568a46d0034-kube-api-access-2rww9\") pod \"payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4\" (UID: \"ee57b283-ffc3-4310-9113-6568a46d0034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:55.381770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.381720 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee57b283-ffc3-4310-9113-6568a46d0034-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4\" (UID: \"ee57b283-ffc3-4310-9113-6568a46d0034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:55.392013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.389336 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee57b283-ffc3-4310-9113-6568a46d0034-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4\" (UID: \"ee57b283-ffc3-4310-9113-6568a46d0034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:55.581126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.581083 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5"] Nov 02 14:39:55.587838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.587813 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:55.588185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.588115 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5"] Nov 02 14:39:55.593243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.593216 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rww9\" (UniqueName: \"kubernetes.io/projected/ee57b283-ffc3-4310-9113-6568a46d0034-kube-api-access-2rww9\") pod \"payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4\" (UID: \"ee57b283-ffc3-4310-9113-6568a46d0034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:55.696120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.696088 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6ct\" (UniqueName: \"kubernetes.io/projected/07dac19e-1503-410b-8eab-6905bb48ec0b-kube-api-access-4s6ct\") pod \"payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5\" (UID: \"07dac19e-1503-410b-8eab-6905bb48ec0b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:55.696289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.696150 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dac19e-1503-410b-8eab-6905bb48ec0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5\" (UID: \"07dac19e-1503-410b-8eab-6905bb48ec0b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:55.739994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.739950 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:55.798477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.798436 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6ct\" (UniqueName: \"kubernetes.io/projected/07dac19e-1503-410b-8eab-6905bb48ec0b-kube-api-access-4s6ct\") pod \"payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5\" (UID: \"07dac19e-1503-410b-8eab-6905bb48ec0b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:55.798994 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.798955 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dac19e-1503-410b-8eab-6905bb48ec0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5\" (UID: \"07dac19e-1503-410b-8eab-6905bb48ec0b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:55.799956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.799931 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dac19e-1503-410b-8eab-6905bb48ec0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5\" (UID: \"07dac19e-1503-410b-8eab-6905bb48ec0b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:55.832737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.832699 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6ct\" (UniqueName: \"kubernetes.io/projected/07dac19e-1503-410b-8eab-6905bb48ec0b-kube-api-access-4s6ct\") pod \"payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5\" (UID: \"07dac19e-1503-410b-8eab-6905bb48ec0b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:55.916764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:55.916729 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:56.111214 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:39:56.108905 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee57b283_ffc3_4310_9113_6568a46d0034.slice/crio-a1406b104a7bf0776cf5c916e9528e08671b9a476240febdf94797486636d700 WatchSource:0}: Error finding container a1406b104a7bf0776cf5c916e9528e08671b9a476240febdf94797486636d700: Status 404 returned error can't find the container with id a1406b104a7bf0776cf5c916e9528e08671b9a476240febdf94797486636d700 Nov 02 14:39:56.134005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.116181 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:56.134005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.119142 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4"] Nov 02 14:39:56.237152 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:39:56.237121 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07dac19e_1503_410b_8eab_6905bb48ec0b.slice/crio-d4fd3b92c1cff6d8aa41437e87fbbd746df3b44b8cb008ae1c6258e4cd9832fd WatchSource:0}: Error finding container d4fd3b92c1cff6d8aa41437e87fbbd746df3b44b8cb008ae1c6258e4cd9832fd: Status 404 returned error can't find the container with id d4fd3b92c1cff6d8aa41437e87fbbd746df3b44b8cb008ae1c6258e4cd9832fd Nov 02 14:39:56.237315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.237292 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5"] Nov 02 14:39:56.864005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.863560 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" event={"ID":"07dac19e-1503-410b-8eab-6905bb48ec0b","Type":"ContainerStarted","Data":"b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107"} Nov 02 14:39:56.864005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.863651 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" event={"ID":"07dac19e-1503-410b-8eab-6905bb48ec0b","Type":"ContainerStarted","Data":"d4fd3b92c1cff6d8aa41437e87fbbd746df3b44b8cb008ae1c6258e4cd9832fd"} Nov 02 14:39:56.868007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.865418 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:39:56.868007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.867642 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" event={"ID":"ee57b283-ffc3-4310-9113-6568a46d0034","Type":"ContainerStarted","Data":"3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39"} Nov 02 14:39:56.868007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.867691 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" event={"ID":"ee57b283-ffc3-4310-9113-6568a46d0034","Type":"ContainerStarted","Data":"a1406b104a7bf0776cf5c916e9528e08671b9a476240febdf94797486636d700"} Nov 02 14:39:56.890009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.868512 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:39:56.942519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.942447 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" podStartSLOduration=1.942425495 podStartE2EDuration="1.942425495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:56.912269706 +0000 UTC m=+3345.637891477" watchObservedRunningTime="2026-11-02 14:39:56.942425495 +0000 UTC m=+3345.668047265" Nov 02 14:39:56.984074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:56.984009 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" podStartSLOduration=1.9839736860000001 podStartE2EDuration="1.983973686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:56.959547971 +0000 UTC m=+3345.685169742" watchObservedRunningTime="2026-11-02 14:39:56.983973686 +0000 UTC m=+3345.709595458" Nov 02 14:39:57.991010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:57.984487 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:39:59.685008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:39:59.679575 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:01.075722 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:40:01.075680 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:40:01.408015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:01.404630 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:03.055636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:03.055544 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:04.735582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:04.735542 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:06.306030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:06.305922 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:07.879787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:07.879753 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:40:07.931549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:07.931511 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:08.887997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:08.887964 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:40:09.583925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:09.578353 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:11.378022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:11.374917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:11.781268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:11.781230 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:11.948351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:11.948316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:13.117851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:13.117809 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:14.806006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:14.799265 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:16.562056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:16.561953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:18.338039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:18.328097 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:20.092705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:20.092657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:21.103477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:21.103435 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh"] Nov 02 14:40:21.103921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:21.103700 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" containerID="cri-o://4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a" gracePeriod=30 Nov 02 14:40:21.650116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:21.650083 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:21.862007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:21.851674 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:40:23.505577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:23.505537 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:23.990659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:23.990620 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6"] Nov 02 14:40:23.990948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:23.990842 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" containerID="cri-o://a40bba18a070b32cea4624f3e8ac8ff2b206be8cf3365837ed635fe6f555dfee" gracePeriod=30 Nov 02 14:40:25.360112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:25.359998 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:26.471778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.471751 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 14:40:26.503533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503499 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-socket\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.503699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503544 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-envoy\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.503699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503599 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-credential-socket\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.503790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503697 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-token\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-token\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.503790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503757 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-kube-api-access-qqk6d\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-kube-api-access-qqk6d\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.503790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503780 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-data\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.503953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503788 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-socket" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:26.503953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503839 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istiod-ca-cert\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.503953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.503861 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-podinfo\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.504182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.504022 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-certs\") pod \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\" (UID: \"8e7d3fe4-0505-4a74-a801-90cd65ea1a41\") " Nov 02 14:40:26.504182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.504026 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-credential-socket" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:26.504182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.504133 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-data" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:26.504365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.504323 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-certs" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:26.504365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.504260 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istiod-ca-cert" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:40:26.505126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.505021 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.505126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.505050 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.505126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.505066 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.505126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.505079 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.505126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.505095 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.506239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.506208 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-envoy" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:26.506363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.506319 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-kube-api-access-qqk6d" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "kube-api-access-qqk6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:26.506363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.506341 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-podinfo" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:40:26.506499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.506380 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-token" pod "8e7d3fe4-0505-4a74-a801-90cd65ea1a41" (UID: "8e7d3fe4-0505-4a74-a801-90cd65ea1a41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:26.606774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.606730 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.606774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.606776 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqk6d\" (UniqueName: \"kubernetes.io/projected/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-kube-api-access-qqk6d\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.607016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.606791 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.607016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.606805 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7d3fe4-0505-4a74-a801-90cd65ea1a41-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:26.670427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:26.670315 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:27.163506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.161740 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:27.223835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.223797 5591 generic.go:356] "Generic (PLEG): container finished" podID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerID="4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a" exitCode=0 Nov 02 14:40:27.224032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.223895 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" event={"ID":"8e7d3fe4-0505-4a74-a801-90cd65ea1a41","Type":"ContainerDied","Data":"4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a"} Nov 02 14:40:27.224032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.223949 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" event={"ID":"8e7d3fe4-0505-4a74-a801-90cd65ea1a41","Type":"ContainerDied","Data":"533b8f2c32b408473d201a8bd6431245fa28c9f1a40bd7d4f8b2380dec831880"} Nov 02 14:40:27.224032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.223972 5591 scope.go:122] "RemoveContainer" containerID="4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a" Nov 02 14:40:27.224163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.224123 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh" Nov 02 14:40:27.271666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.271643 5591 scope.go:122] "RemoveContainer" containerID="fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03" Nov 02 14:40:27.282426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.282402 5591 scope.go:122] "RemoveContainer" containerID="4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a" Nov 02 14:40:27.282823 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:40:27.282797 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a\": container with ID starting with 4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a not found: ID does not exist" containerID="4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a" Nov 02 14:40:27.282887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.282834 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a"} err="failed to get container status \"4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a\": rpc error: code = NotFound desc = could not find container \"4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a\": container with ID starting with 4519667ef602601ddb3032a5629e406c828e4bc89329ac3ae445b666c7cb2e5a not found: ID does not exist" Nov 02 14:40:27.282887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.282853 5591 scope.go:122] "RemoveContainer" containerID="fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03" Nov 02 14:40:27.283138 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:40:27.283113 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03\": container with ID starting with fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03 not found: ID does not exist" containerID="fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03" Nov 02 14:40:27.283221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.283147 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03"} err="failed to get container status \"fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03\": rpc error: code = NotFound desc = could not find container \"fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03\": container with ID starting with fd705e0c13447d9f0489b047a974c7254a14ebcfa01689ed229c12c05dd82c03 not found: ID does not exist" Nov 02 14:40:27.311423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.311381 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh"] Nov 02 14:40:27.324694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.324664 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s6gjh"] Nov 02 14:40:27.458837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:27.458751 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" path="/var/lib/kubelet/pods/8e7d3fe4-0505-4a74-a801-90cd65ea1a41/volumes" Nov 02 14:40:28.961050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:28.959622 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:29.270823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.270726 5591 generic.go:356] "Generic (PLEG): container finished" podID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerID="a40bba18a070b32cea4624f3e8ac8ff2b206be8cf3365837ed635fe6f555dfee" exitCode=0 Nov 02 14:40:29.270823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.270773 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" event={"ID":"09eab821-d26d-4bde-addc-f46da6c42ac8","Type":"ContainerDied","Data":"a40bba18a070b32cea4624f3e8ac8ff2b206be8cf3365837ed635fe6f555dfee"} Nov 02 14:40:29.310867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.310839 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:40:29.383185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383145 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-credential-socket\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.383372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383276 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-envoy\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.383372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383308 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-kube-api-access-xfgb6\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-kube-api-access-xfgb6\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.383549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383450 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-podinfo\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.383549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383510 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09eab821-d26d-4bde-addc-f46da6c42ac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eab821-d26d-4bde-addc-f46da6c42ac8-istiod-ca-cert\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.383707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383536 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-credential-socket" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:29.383707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383585 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-socket\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.383813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383773 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-token\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-token\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.383920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383904 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-data\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.384047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.383962 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-certs\") pod \"09eab821-d26d-4bde-addc-f46da6c42ac8\" (UID: \"09eab821-d26d-4bde-addc-f46da6c42ac8\") " Nov 02 14:40:29.384047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384035 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09eab821-d26d-4bde-addc-f46da6c42ac8-istiod-ca-cert" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:40:29.384760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384259 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-data" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:29.384760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384447 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-socket" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:29.384760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384457 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-certs" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:29.385110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384844 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:29.385110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384866 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:29.385110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384880 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:29.385110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384895 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09eab821-d26d-4bde-addc-f46da6c42ac8-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:29.385110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.384914 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:29.401004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.400414 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-envoy" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:29.404000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.403525 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-podinfo" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:40:29.411329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.410303 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-token" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:29.411329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.410747 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-kube-api-access-xfgb6" pod "09eab821-d26d-4bde-addc-f46da6c42ac8" (UID: "09eab821-d26d-4bde-addc-f46da6c42ac8"). InnerVolumeSpecName "kube-api-access-xfgb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:29.493635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.493599 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:29.493635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.493632 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfgb6\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-kube-api-access-xfgb6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:29.493885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.493646 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:29.493885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:29.493660 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09eab821-d26d-4bde-addc-f46da6c42ac8-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:30.282070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:30.282035 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" Nov 02 14:40:30.282541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:30.282024 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6" event={"ID":"09eab821-d26d-4bde-addc-f46da6c42ac8","Type":"ContainerDied","Data":"190f43f1ea0213f99f989b01ce2ffba6d546455ab3bb749dbc3fbb04999146d6"} Nov 02 14:40:30.282541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:30.282192 5591 scope.go:122] "RemoveContainer" containerID="a40bba18a070b32cea4624f3e8ac8ff2b206be8cf3365837ed635fe6f555dfee" Nov 02 14:40:30.357976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:30.357936 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6"] Nov 02 14:40:30.368003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:30.367953 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rqcb6"] Nov 02 14:40:30.796901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:30.796681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:31.081684 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:40:31.081599 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:40:31.434649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:31.434608 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" path="/var/lib/kubelet/pods/09eab821-d26d-4bde-addc-f46da6c42ac8/volumes" Nov 02 14:40:32.486008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:32.475608 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:34.177651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:34.173265 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:34.320348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:34.320317 5591 generic.go:356] "Generic (PLEG): container finished" podID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerID="6b8fc3c7e0e74e5ee876768bb8822f005f6c476adab14f612ed9292e07cc8a7d" exitCode=0 Nov 02 14:40:34.320535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:34.320393 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" event={"ID":"bd282e44-172b-4b31-81ce-ddfc8fd63a78","Type":"ContainerDied","Data":"6b8fc3c7e0e74e5ee876768bb8822f005f6c476adab14f612ed9292e07cc8a7d"} Nov 02 14:40:34.320875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:34.320858 5591 scope.go:122] "RemoveContainer" containerID="6b8fc3c7e0e74e5ee876768bb8822f005f6c476adab14f612ed9292e07cc8a7d" Nov 02 14:40:34.323013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:34.322952 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:40:34.323117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:34.323045 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:40:34.323117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:34.323080 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:40:35.361273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:35.355682 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" event={"ID":"bd282e44-172b-4b31-81ce-ddfc8fd63a78","Type":"ContainerStarted","Data":"56416d2f8eaa7f5768a647c01f49e347ec40083d42f5f351267526c541087092"} Nov 02 14:40:35.967014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:35.966425 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:36.283486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:36.283397 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:40:36.285165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:36.285134 5591 patch_prober.go:29] interesting pod/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 02 14:40:36.285306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:36.285196 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 02 14:40:37.284465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:37.284426 5591 patch_prober.go:29] interesting pod/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 02 14:40:37.284902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:37.284491 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 02 14:40:37.416538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:37.416414 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:37.783081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:37.783034 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:37.951510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:37.951456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:38.265016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.264956 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt"] Nov 02 14:40:38.266296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266270 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" Nov 02 14:40:38.266296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266296 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" Nov 02 14:40:38.266455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266315 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" Nov 02 14:40:38.266455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266320 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" Nov 02 14:40:38.266455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266333 5591 cpu_manager.go:414] "containerMap: removing container" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" Nov 02 14:40:38.266455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266338 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" Nov 02 14:40:38.266593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266481 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" Nov 02 14:40:38.266593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266489 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7d3fe4-0505-4a74-a801-90cd65ea1a41" containerName="istio-proxy" Nov 02 14:40:38.266593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.266497 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="09eab821-d26d-4bde-addc-f46da6c42ac8" containerName="istio-proxy" Nov 02 14:40:38.269613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.269593 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.284184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.284152 5591 patch_prober.go:29] interesting pod/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 02 14:40:38.284308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.284208 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 02 14:40:38.298951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.298916 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt"] Nov 02 14:40:38.308542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.308512 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-80133f-openshift-default-dockercfg-fn4tq\"" Nov 02 14:40:38.375062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.374874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.404735 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-credential-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.404900 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-istio-token\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.404943 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dede1-2042-4f22-8415-4e9497e0b26b-istiod-ca-cert\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.404994 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-envoy\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.405124 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-certs\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.405218 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-data\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.405269 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4fd4\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-kube-api-access-h4fd4\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.405304 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dede1-2042-4f22-8415-4e9497e0b26b-istio-podinfo\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.412035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.405362 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.520192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.520095 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-credential-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.520588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.520561 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-istio-token\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.520927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.520903 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dede1-2042-4f22-8415-4e9497e0b26b-istiod-ca-cert\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.521096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.521078 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-envoy\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.521424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.521403 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-certs\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.521619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.521601 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-data\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.521775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.521758 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4fd4\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-kube-api-access-h4fd4\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.521904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.521888 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dede1-2042-4f22-8415-4e9497e0b26b-istio-podinfo\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.522137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.522052 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.522945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.522921 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.523472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.523449 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-credential-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.524589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.524568 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dede1-2042-4f22-8415-4e9497e0b26b-istiod-ca-cert\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.526111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.525588 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-data\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.562028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.559603 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-certs\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.569027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.565718 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:38.595019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.592461 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-envoy\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.613016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.611523 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4fd4\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-kube-api-access-h4fd4\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.702008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.693820 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dede1-2042-4f22-8415-4e9497e0b26b-istio-podinfo\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.716009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.702861 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-istio-token\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:38.884197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:38.884169 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:39.059036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.055885 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh"] Nov 02 14:40:39.063497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.063453 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" containerID="cri-o://597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423" gracePeriod=30 Nov 02 14:40:39.259927 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:40:39.259894 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874dede1_2042_4f22_8415_4e9497e0b26b.slice/crio-31bbe0e1d499eb8a7877e64e3d0fea13b197e6d306a29e2fff086fbe706580c6 WatchSource:0}: Error finding container 31bbe0e1d499eb8a7877e64e3d0fea13b197e6d306a29e2fff086fbe706580c6: Status 404 returned error can't find the container with id 31bbe0e1d499eb8a7877e64e3d0fea13b197e6d306a29e2fff086fbe706580c6 Nov 02 14:40:39.262676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.262650 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt"] Nov 02 14:40:39.262766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.262739 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:40:39.262815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.262805 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:40:39.262856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.262836 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:40:39.283764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.283739 5591 patch_prober.go:29] interesting pod/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 02 14:40:39.283875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.283796 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 02 14:40:39.689156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.681961 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:39.705832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.705782 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" event={"ID":"874dede1-2042-4f22-8415-4e9497e0b26b","Type":"ContainerStarted","Data":"366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d"} Nov 02 14:40:39.705832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.705838 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" event={"ID":"874dede1-2042-4f22-8415-4e9497e0b26b","Type":"ContainerStarted","Data":"31bbe0e1d499eb8a7877e64e3d0fea13b197e6d306a29e2fff086fbe706580c6"} Nov 02 14:40:39.755007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.754498 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" podStartSLOduration=1.7544783069999998 podStartE2EDuration="1.754478307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:40:39.752283422 +0000 UTC m=+3388.477905196" watchObservedRunningTime="2026-11-02 14:40:39.754478307 +0000 UTC m=+3388.480100081" Nov 02 14:40:39.762088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.757066 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:40:39.888317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:39.885111 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:40.284455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:40.284416 5591 patch_prober.go:29] interesting pod/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 02 14:40:40.284650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:40.284485 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 02 14:40:40.891243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:40.891207 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:41.326065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:41.315376 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:40:41.359010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:41.351531 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:40:41.611038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:41.610898 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:41.723890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:41.723851 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:41.724755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:41.724730 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:40:42.303786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:42.303727 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:42.487674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:42.487637 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:43.517537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:43.517495 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:43.984628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:43.984577 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:44.050976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.050937 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g"] Nov 02 14:40:44.055035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.054969 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:44.088832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.088743 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-80133f-dockercfg-5c94d\"" Nov 02 14:40:44.089058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.088853 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-80133f\"" Nov 02 14:40:44.094484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.094450 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g"] Nov 02 14:40:44.098780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.098744 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhgf\" (UniqueName: \"kubernetes.io/projected/a8cef1df-75f5-4a87-970e-257e87108d9b-kube-api-access-xbhgf\") pod \"payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g\" (UID: \"a8cef1df-75f5-4a87-970e-257e87108d9b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:44.099104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.099061 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cef1df-75f5-4a87-970e-257e87108d9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g\" (UID: \"a8cef1df-75f5-4a87-970e-257e87108d9b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:44.106324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.106294 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69"] Nov 02 14:40:44.111235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.111216 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:44.145297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.145263 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69"] Nov 02 14:40:44.172907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.172860 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:44.200853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.200793 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7vch\" (UniqueName: \"kubernetes.io/projected/53af4c44-b710-4d7f-a167-d86391499b3c-kube-api-access-h7vch\") pod \"payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69\" (UID: \"53af4c44-b710-4d7f-a167-d86391499b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:44.201036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.200965 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53af4c44-b710-4d7f-a167-d86391499b3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69\" (UID: \"53af4c44-b710-4d7f-a167-d86391499b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:44.201100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.201072 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhgf\" (UniqueName: \"kubernetes.io/projected/a8cef1df-75f5-4a87-970e-257e87108d9b-kube-api-access-xbhgf\") pod \"payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g\" (UID: \"a8cef1df-75f5-4a87-970e-257e87108d9b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:44.201227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.201207 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cef1df-75f5-4a87-970e-257e87108d9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g\" (UID: \"a8cef1df-75f5-4a87-970e-257e87108d9b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:44.201935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.201913 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cef1df-75f5-4a87-970e-257e87108d9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g\" (UID: \"a8cef1df-75f5-4a87-970e-257e87108d9b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:44.230157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.230123 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhgf\" (UniqueName: \"kubernetes.io/projected/a8cef1df-75f5-4a87-970e-257e87108d9b-kube-api-access-xbhgf\") pod \"payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g\" (UID: \"a8cef1df-75f5-4a87-970e-257e87108d9b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:44.368018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.366484 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:44.452003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.444943 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53af4c44-b710-4d7f-a167-d86391499b3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69\" (UID: \"53af4c44-b710-4d7f-a167-d86391499b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:44.452003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.445316 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7vch\" (UniqueName: \"kubernetes.io/projected/53af4c44-b710-4d7f-a167-d86391499b3c-kube-api-access-h7vch\") pod \"payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69\" (UID: \"53af4c44-b710-4d7f-a167-d86391499b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:44.452003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.445789 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:44.566133 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.456141 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53af4c44-b710-4d7f-a167-d86391499b3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69\" (UID: \"53af4c44-b710-4d7f-a167-d86391499b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:44.594485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.586930 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7vch\" (UniqueName: \"kubernetes.io/projected/53af4c44-b710-4d7f-a167-d86391499b3c-kube-api-access-h7vch\") pod \"payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69\" (UID: \"53af4c44-b710-4d7f-a167-d86391499b3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:44.667700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.667668 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:40:44.751169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.751132 5591 generic.go:356] "Generic (PLEG): container finished" podID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerID="597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423" exitCode=0 Nov 02 14:40:44.751393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.751191 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" Nov 02 14:40:44.751393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.751222 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" event={"ID":"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1","Type":"ContainerDied","Data":"597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423"} Nov 02 14:40:44.751393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.751289 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh" event={"ID":"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1","Type":"ContainerDied","Data":"54ba52e9b67c55b4ae54de633b81b975016f4efb7d3ac02e9ddb7db111544a09"} Nov 02 14:40:44.751393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.751321 5591 scope.go:122] "RemoveContainer" containerID="597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423" Nov 02 14:40:44.763762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.763739 5591 scope.go:122] "RemoveContainer" containerID="597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423" Nov 02 14:40:44.767656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.767597 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:44.767814 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:40:44.767734 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423\": container with ID starting with 597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423 not found: ID does not exist" containerID="597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423" Nov 02 14:40:44.767814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.767774 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423"} err="failed to get container status \"597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423\": rpc error: code = NotFound desc = could not find container \"597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423\": container with ID starting with 597312177f42c373ab0a2d1afefedaae3770762f120456cfe5d63dba25164423 not found: ID does not exist" Nov 02 14:40:44.785400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.785375 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-kube-api-access-7r2xn\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-kube-api-access-7r2xn\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.785512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.785464 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-credential-socket\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.785512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.785499 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-envoy\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.785639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.785607 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-podinfo\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.785698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.785643 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-data\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.785829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.785814 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-token\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-token\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.786092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.785886 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-certs\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.786092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.785922 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-socket\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.786337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.786261 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-socket" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:44.786337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.786278 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istiod-ca-cert\") pod \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\" (UID: \"26ea71ac-fe5a-4018-95f7-6a71a4bec2a1\") " Nov 02 14:40:44.786337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.786289 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-data" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:44.786533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.786443 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-credential-socket" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:44.786696 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.786677 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-certs" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:44.786947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.786927 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istiod-ca-cert" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:40:44.787327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.787304 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.787404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.787333 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.787404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.787349 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.787404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.787363 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.787404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.787378 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.789280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.789250 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-token" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:44.790137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.790107 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-podinfo" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:40:44.795479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.795453 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-envoy" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:44.796169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.796145 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-kube-api-access-7r2xn" pod "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" (UID: "26ea71ac-fe5a-4018-95f7-6a71a4bec2a1"). InnerVolumeSpecName "kube-api-access-7r2xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:44.889657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.889178 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r2xn\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-kube-api-access-7r2xn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.889657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.889213 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.889657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.889231 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.889657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.889243 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:40:44.949242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:44.949161 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g"] Nov 02 14:40:44.949506 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:40:44.949476 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cef1df_75f5_4a87_970e_257e87108d9b.slice/crio-9bd0810b3b6083e4d3136904ba266addc0e2124af486a6d8b78d8e7ca9efabcf WatchSource:0}: Error finding container 9bd0810b3b6083e4d3136904ba266addc0e2124af486a6d8b78d8e7ca9efabcf: Status 404 returned error can't find the container with id 9bd0810b3b6083e4d3136904ba266addc0e2124af486a6d8b78d8e7ca9efabcf Nov 02 14:40:45.090523 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:40:45.090493 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53af4c44_b710_4d7f_a167_d86391499b3c.slice/crio-def64c18f39e879fa4c1a8008391f1fa8cc78c499086697b050511662e53e315 WatchSource:0}: Error finding container def64c18f39e879fa4c1a8008391f1fa8cc78c499086697b050511662e53e315: Status 404 returned error can't find the container with id def64c18f39e879fa4c1a8008391f1fa8cc78c499086697b050511662e53e315 Nov 02 14:40:45.090644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.090615 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69"] Nov 02 14:40:45.108617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.108582 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh"] Nov 02 14:40:45.113396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.113374 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-77nsh"] Nov 02 14:40:45.426422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.426390 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:45.438352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.438320 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" path="/var/lib/kubelet/pods/26ea71ac-fe5a-4018-95f7-6a71a4bec2a1/volumes" Nov 02 14:40:45.762595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.762496 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" event={"ID":"a8cef1df-75f5-4a87-970e-257e87108d9b","Type":"ContainerStarted","Data":"04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9"} Nov 02 14:40:45.762595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.762569 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:45.762595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.762593 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" event={"ID":"a8cef1df-75f5-4a87-970e-257e87108d9b","Type":"ContainerStarted","Data":"9bd0810b3b6083e4d3136904ba266addc0e2124af486a6d8b78d8e7ca9efabcf"} Nov 02 14:40:45.766368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.766333 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" event={"ID":"53af4c44-b710-4d7f-a167-d86391499b3c","Type":"ContainerStarted","Data":"bcc5c2182ea113bcaaf541ffe610b817703950150c5c7ab53417a550e97588a6"} Nov 02 14:40:45.766505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.766391 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:45.766505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.766410 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" event={"ID":"53af4c44-b710-4d7f-a167-d86391499b3c","Type":"ContainerStarted","Data":"def64c18f39e879fa4c1a8008391f1fa8cc78c499086697b050511662e53e315"} Nov 02 14:40:45.848536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.848485 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" podStartSLOduration=1.848471612 podStartE2EDuration="1.848471612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:40:45.843575733 +0000 UTC m=+3394.569197502" watchObservedRunningTime="2026-11-02 14:40:45.848471612 +0000 UTC m=+3394.574093381" Nov 02 14:40:45.911648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.911582 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" podStartSLOduration=1.911562821 podStartE2EDuration="1.911562821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:40:45.908366997 +0000 UTC m=+3394.633988779" watchObservedRunningTime="2026-11-02 14:40:45.911562821 +0000 UTC m=+3394.637184595" Nov 02 14:40:45.990488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.990446 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g"] Nov 02 14:40:45.991401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.991371 5591 cpu_manager.go:414] "containerMap: removing container" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" Nov 02 14:40:45.991401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.991400 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" Nov 02 14:40:45.991637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.991621 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ea71ac-fe5a-4018-95f7-6a71a4bec2a1" containerName="istio-proxy" Nov 02 14:40:45.995599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:45.995580 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.013016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.012909 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fe1881\"/\"kube-root-ca.crt\"" Nov 02 14:40:46.013279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.013256 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fe1881\"/\"openshift-service-ca.crt\"" Nov 02 14:40:46.013435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.013412 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fe1881\"/\"default-dockercfg-khfrq\"" Nov 02 14:40:46.013550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.013533 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fe1881\"/\"test-model-fe1881-kserve-self-signed-certs\"" Nov 02 14:40:46.013701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.013671 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-model-cache\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.013827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.013803 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70742c94-86ad-4a7f-8203-e22a7641366c-tls-certs\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.013947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.013876 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-home\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.014122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.014101 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2ds\" (UniqueName: \"kubernetes.io/projected/70742c94-86ad-4a7f-8203-e22a7641366c-kube-api-access-gs2ds\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.014241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.014222 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-tmp-dir\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.014431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.014413 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-dshm\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.018912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.018889 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g"] Nov 02 14:40:46.121279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.121236 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70742c94-86ad-4a7f-8203-e22a7641366c-tls-certs\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.121500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.121292 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-home\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.121500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.121457 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2ds\" (UniqueName: \"kubernetes.io/projected/70742c94-86ad-4a7f-8203-e22a7641366c-kube-api-access-gs2ds\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.121624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.121539 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-tmp-dir\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.121702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.121685 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-dshm\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.121777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.121752 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-home\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.121830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.121759 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-model-cache\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.122155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.122132 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-model-cache\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.122262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.122243 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-tmp-dir\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.124201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.124174 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-dshm\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.124201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.124194 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70742c94-86ad-4a7f-8203-e22a7641366c-tls-certs\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.145929 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.145901 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2ds\" (UniqueName: \"kubernetes.io/projected/70742c94-86ad-4a7f-8203-e22a7641366c-kube-api-access-gs2ds\") pod \"test-model-fe1881-kserve-785c68677c-z479g\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.178180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.178151 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:46.320060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.311334 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:46.699366 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:40:46.699329 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70742c94_86ad_4a7f_8203_e22a7641366c.slice/crio-a7618744c9736d538e86668f158474c04e3b3fb37dc711d359d1e35fa5fbf420 WatchSource:0}: Error finding container a7618744c9736d538e86668f158474c04e3b3fb37dc711d359d1e35fa5fbf420: Status 404 returned error can't find the container with id a7618744c9736d538e86668f158474c04e3b3fb37dc711d359d1e35fa5fbf420 Nov 02 14:40:46.700575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.700550 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g"] Nov 02 14:40:46.780140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.779355 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" event={"ID":"70742c94-86ad-4a7f-8203-e22a7641366c","Type":"ContainerStarted","Data":"d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae"} Nov 02 14:40:46.780140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.779434 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" event={"ID":"70742c94-86ad-4a7f-8203-e22a7641366c","Type":"ContainerStarted","Data":"a7618744c9736d538e86668f158474c04e3b3fb37dc711d359d1e35fa5fbf420"} Nov 02 14:40:46.844423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:46.844365 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" podStartSLOduration=1.8443515480000001 podStartE2EDuration="1.844351548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:40:46.838915636 +0000 UTC m=+3395.564537407" watchObservedRunningTime="2026-11-02 14:40:46.844351548 +0000 UTC m=+3395.569973317" Nov 02 14:40:47.312502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:47.312471 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:47.381622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:47.381588 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:49.081198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:49.075661 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:50.881537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:50.881445 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:52.660802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:52.660767 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:52.839788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:52.839727 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:54.665819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:54.665780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:56.311729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:56.311688 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:56.325254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:56.325227 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:56.584685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:56.584568 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:40:56.782594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:56.782563 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:40:56.782800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:56.782644 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:40:56.961251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:56.961219 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:40:58.309489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:40:58.309455 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:00.207203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:00.207166 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:01.076361 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:41:01.076324 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:41:02.041659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:02.038719 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:03.814661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:03.814624 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:05.610007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:05.597154 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:07.467426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:07.467334 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:09.406006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:09.402648 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:09.779492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:09.779406 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:11.278179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:11.278142 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:13.077012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:13.075315 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:14.969150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:14.969113 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:15.548005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:15.536077 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:41:15.681855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:15.681816 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:16.829501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:16.829464 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:18.741712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:18.741560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:20.853024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:20.849584 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:22.814062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:22.813171 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:24.561690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:24.561607 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:26.505527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:26.505488 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:26.791248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:26.791218 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:27.295773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:27.295735 5591 generic.go:356] "Generic (PLEG): container finished" podID="91593d10-d549-4bf5-a212-0a2afb872125" containerID="de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75" exitCode=0 Nov 02 14:41:27.295959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:27.295806 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" event={"ID":"91593d10-d549-4bf5-a212-0a2afb872125","Type":"ContainerDied","Data":"de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75"} Nov 02 14:41:27.296372 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:27.296356 5591 scope.go:122] "RemoveContainer" containerID="de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75" Nov 02 14:41:27.298629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:27.298604 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:41:27.298700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:27.298673 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:41:27.298738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:27.298704 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:41:28.249249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:28.249192 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:28.333182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:28.333145 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" event={"ID":"91593d10-d549-4bf5-a212-0a2afb872125","Type":"ContainerStarted","Data":"e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21"} Nov 02 14:41:29.065179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:29.065145 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:41:29.065396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:29.065349 5591 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 02 14:41:29.065531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:29.065417 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 02 14:41:30.030258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:30.030193 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:30.067397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:30.067350 5591 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 02 14:41:30.067690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:30.067659 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 02 14:41:31.065381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:31.065337 5591 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 02 14:41:31.065824 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:31.065409 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 02 14:41:31.085557 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:41:31.085522 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:41:32.020007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:32.019025 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:32.065506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:32.065463 5591 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 02 14:41:32.066074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:32.065535 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 02 14:41:33.065075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:33.065035 5591 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 02 14:41:33.065279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:33.065097 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 02 14:41:33.811830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:33.811137 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:34.069894 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:34.069808 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:41:34.070363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:34.070341 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:41:34.071055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:34.071033 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:41:35.656840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:35.656804 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:36.970721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:36.970685 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:37.429540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:37.429460 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:39.242056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:39.237635 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:42.205245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:42.205187 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:43.074695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:43.074647 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:44.816104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:44.816067 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:46.605966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:46.605932 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:46.796060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:46.796015 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:48.669085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:48.669049 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:50.593460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:50.593409 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:52.511811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:52.511774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:54.310587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:54.310549 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:56.191172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:56.191131 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:41:58.128162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:41:58.128127 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:00.296946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:00.296890 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:01.077134 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:42:01.077097 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:42:02.299014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:02.298953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:04.248940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:04.248900 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:06.230838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:06.230805 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:06.447244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:06.447191 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:07.790413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:07.790378 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:08.302422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:08.302390 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:10.267005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:10.266219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:10.559862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:10.559778 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj"] Nov 02 14:42:10.560102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:10.560061 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" containerID="cri-o://0638431cf0826a37d3423daccd4c6cd8bbe7e70330e3fa525f4994e5d1792c15" gracePeriod=30 Nov 02 14:42:12.048625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:12.048560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:13.811037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:13.810948 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:15.508264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.508210 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:15.837901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.837859 5591 generic.go:356] "Generic (PLEG): container finished" podID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerID="0638431cf0826a37d3423daccd4c6cd8bbe7e70330e3fa525f4994e5d1792c15" exitCode=0 Nov 02 14:42:15.838090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.837936 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" event={"ID":"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b","Type":"ContainerDied","Data":"0638431cf0826a37d3423daccd4c6cd8bbe7e70330e3fa525f4994e5d1792c15"} Nov 02 14:42:15.838090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.838014 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" event={"ID":"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b","Type":"ContainerDied","Data":"98f0c6ac5bca515168184a9df2245e16710f425c7b798e3b44ef89a61b579e72"} Nov 02 14:42:15.838090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.838042 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f0c6ac5bca515168184a9df2245e16710f425c7b798e3b44ef89a61b579e72" Nov 02 14:42:15.851928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.851897 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:42:15.990404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990362 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-token\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-token\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.990613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990492 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istiod-ca-cert\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.990613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990587 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-podinfo\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.990746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990647 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-envoy\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.990746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990734 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-kube-api-access-bd24n\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-kube-api-access-bd24n\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.990856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990774 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-socket\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.990916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990856 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-data\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.990977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990940 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-credential-socket\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.991054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.990961 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istiod-ca-cert" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:15.991111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.991080 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-certs\") pod \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\" (UID: \"fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b\") " Nov 02 14:42:15.991174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.991123 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-socket" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:15.991600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.991298 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-data" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:15.991600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.991373 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-credential-socket" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:15.991600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.991541 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-certs" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:15.992063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.992020 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:15.992063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.992045 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:15.992225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.992062 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:15.992225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.992080 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:15.992225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.992091 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:15.994095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.994065 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-envoy" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:15.994725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.994681 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-podinfo" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:42:15.994855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.994822 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-kube-api-access-bd24n" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "kube-api-access-bd24n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:15.995111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:15.995086 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-token" pod "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" (UID: "fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:16.093738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:16.093652 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:16.093738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:16.093683 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:16.093738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:16.093693 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:16.093738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:16.093703 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd24n\" (UniqueName: \"kubernetes.io/projected/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b-kube-api-access-bd24n\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:42:16.845403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:16.845369 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj" Nov 02 14:42:16.918762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:16.918725 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj"] Nov 02 14:42:16.930353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:16.930325 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-vhtsj"] Nov 02 14:42:17.283067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:17.277634 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:17.444232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:17.444197 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" path="/var/lib/kubelet/pods/fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b/volumes" Nov 02 14:42:19.186416 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:19.184705 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:20.871112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:20.871078 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:22.705143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:22.705059 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:24.445655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:24.445604 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:26.496036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:26.495975 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:27.001775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:27.001735 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:28.363325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:28.363282 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:28.760018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:28.757330 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:42:30.327281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:30.326282 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:31.101427 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:42:31.101384 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:42:32.187874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:32.187815 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:34.011049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.010056 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:34.765691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.765643 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp"] Nov 02 14:42:34.766710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.766683 5591 cpu_manager.go:414] "containerMap: removing container" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" Nov 02 14:42:34.766710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.766707 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" Nov 02 14:42:34.766939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.766871 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde3a69e-c4ea-48ec-b6ef-ceb1b1d7070b" containerName="istio-proxy" Nov 02 14:42:34.772381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.772359 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.779365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.779337 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tmp-dir\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.779574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.779551 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-home\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.779663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.779602 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-dshm\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.779663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.779647 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lqhd\" (UniqueName: \"kubernetes.io/projected/d6d769f2-5c3b-4c9b-9795-df1032663fe6-kube-api-access-2lqhd\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.779789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.779722 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tls-certs\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.779789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.779758 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-model-cache\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.806078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.806039 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp"] Nov 02 14:42:34.811862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.811821 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ea3e8f\"/\"kube-root-ca.crt\"" Nov 02 14:42:34.812047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.811971 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ea3e8f\"/\"openshift-service-ca.crt\"" Nov 02 14:42:34.812188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.812167 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ea3e8f\"/\"test-model-ea3e8f-kserve-self-signed-certs\"" Nov 02 14:42:34.818755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.818723 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ea3e8f\"/\"default-dockercfg-gkshj\"" Nov 02 14:42:34.914016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.913114 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tls-certs\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.914016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.913192 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-model-cache\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.914016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.913360 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tmp-dir\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.914016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.913611 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-home\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.914016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.913655 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-dshm\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.914016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.913709 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lqhd\" (UniqueName: \"kubernetes.io/projected/d6d769f2-5c3b-4c9b-9795-df1032663fe6-kube-api-access-2lqhd\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.930361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.926703 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-home\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.930361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.927034 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-model-cache\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.946039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.939353 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tmp-dir\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.984270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.983673 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lqhd\" (UniqueName: \"kubernetes.io/projected/d6d769f2-5c3b-4c9b-9795-df1032663fe6-kube-api-access-2lqhd\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.991348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.991302 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-dshm\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:34.995167 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:34.995142 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tls-certs\") pod \"test-model-ea3e8f-kserve-7c8c77cc59-pm7pp\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:35.092451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:35.092362 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:35.101660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:35.101622 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:35.696040 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:42:35.695968 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6d769f2_5c3b_4c9b_9795_df1032663fe6.slice/crio-609cc63d8808753b184e24a38f8c697151d15d321f348bff922fc867a6135851 WatchSource:0}: Error finding container 609cc63d8808753b184e24a38f8c697151d15d321f348bff922fc867a6135851: Status 404 returned error can't find the container with id 609cc63d8808753b184e24a38f8c697151d15d321f348bff922fc867a6135851 Nov 02 14:42:35.696527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:35.696413 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp"] Nov 02 14:42:36.065845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:36.065749 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" event={"ID":"d6d769f2-5c3b-4c9b-9795-df1032663fe6","Type":"ContainerStarted","Data":"6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7"} Nov 02 14:42:36.065845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:36.065801 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" event={"ID":"d6d769f2-5c3b-4c9b-9795-df1032663fe6","Type":"ContainerStarted","Data":"609cc63d8808753b184e24a38f8c697151d15d321f348bff922fc867a6135851"} Nov 02 14:42:36.092825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:36.092794 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:36.156957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:36.156904 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" podStartSLOduration=2.156889233 podStartE2EDuration="2.156889233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:42:36.156570201 +0000 UTC m=+3504.882191972" watchObservedRunningTime="2026-11-02 14:42:36.156889233 +0000 UTC m=+3504.882511003" Nov 02 14:42:36.313079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:36.311128 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:38.114014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:38.111358 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:39.888008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:39.885153 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:41.152064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.152029 5591 generic.go:356] "Generic (PLEG): container finished" podID="874dede1-2042-4f22-8415-4e9497e0b26b" containerID="366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d" exitCode=0 Nov 02 14:42:41.152530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.152110 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" event={"ID":"874dede1-2042-4f22-8415-4e9497e0b26b","Type":"ContainerDied","Data":"366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d"} Nov 02 14:42:41.152637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.152622 5591 scope.go:122] "RemoveContainer" containerID="366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d" Nov 02 14:42:41.155022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.154978 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:42:41.155136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.155060 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:42:41.155136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.155114 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:42:41.559065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.555170 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:41.723637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.723600 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:42:41.860750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:41.860672 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:42.066630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:42.066593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:42.168078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:42.168036 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" event={"ID":"874dede1-2042-4f22-8415-4e9497e0b26b","Type":"ContainerStarted","Data":"01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5"} Nov 02 14:42:42.885055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:42.885024 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:42:42.886146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:42.886118 5591 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 14:42:42.886267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:42.886176 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 14:42:43.459791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:43.459386 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:43.885312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:43.885279 5591 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 14:42:43.885514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:43.885364 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 14:42:44.885263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:44.885226 5591 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 14:42:44.885682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:44.885290 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 14:42:45.093291 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:45.093256 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:45.141881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:45.141770 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:45.185387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:45.185354 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:45.209813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:45.209780 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:42:45.885145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:45.885112 5591 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 14:42:45.885345 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:45.885179 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 14:42:46.884672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:46.884633 5591 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 14:42:46.884880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:46.884696 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 14:42:47.012027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:47.011479 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:47.890427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:47.890396 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:42:47.890889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:47.890869 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:42:47.891567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:47.891534 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:42:48.073689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:48.073653 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:48.817468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:48.817429 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:50.770572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:50.770540 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:52.604070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:52.603977 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:54.353629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:54.353545 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:56.323765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:56.323511 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:58.193847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:58.185215 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:42:59.977419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:42:59.976518 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:00.875436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:00.875401 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:01.076816 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:43:01.076776 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:43:02.011937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.011905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:02.458100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.458067 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89"] Nov 02 14:43:02.463053 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.463030 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.495610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.495563 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c8f647\"/\"openshift-service-ca.crt\"" Nov 02 14:43:02.495808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.495614 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c8f647\"/\"default-dockercfg-7cgwv\"" Nov 02 14:43:02.495808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.495679 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c8f647\"/\"test-model-c8f647-kserve-self-signed-certs\"" Nov 02 14:43:02.501197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.501164 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c8f647\"/\"kube-root-ca.crt\"" Nov 02 14:43:02.502682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.502652 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89"] Nov 02 14:43:02.616647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.616609 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-tmp-dir\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.616863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.616690 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-model-cache\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.616950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.616878 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c782b989-1334-4b2d-a7dd-1080b0951ea7-tls-certs\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.616950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.616929 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-dshm\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.617196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.617167 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpp8\" (UniqueName: \"kubernetes.io/projected/c782b989-1334-4b2d-a7dd-1080b0951ea7-kube-api-access-wdpp8\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.617321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.617219 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-home\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.632757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.632728 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:02.732057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.720236 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpp8\" (UniqueName: \"kubernetes.io/projected/c782b989-1334-4b2d-a7dd-1080b0951ea7-kube-api-access-wdpp8\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.732057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.720320 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-home\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.732057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.720472 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-tmp-dir\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.732057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.720559 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-model-cache\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.732057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.720693 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c782b989-1334-4b2d-a7dd-1080b0951ea7-tls-certs\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.732057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.720726 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-dshm\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.742905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.742871 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-tmp-dir\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.744280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.744245 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-dshm\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.752009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.747171 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-model-cache\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.752421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.752364 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-home\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.754006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.753578 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c782b989-1334-4b2d-a7dd-1080b0951ea7-tls-certs\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.761814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.761788 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpp8\" (UniqueName: \"kubernetes.io/projected/c782b989-1334-4b2d-a7dd-1080b0951ea7-kube-api-access-wdpp8\") pod \"test-model-c8f647-kserve-7d874f787f-7pz89\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:02.794017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:02.790311 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:03.203685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:03.202830 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89"] Nov 02 14:43:03.386265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:03.386228 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" event={"ID":"c782b989-1334-4b2d-a7dd-1080b0951ea7","Type":"ContainerStarted","Data":"d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6"} Nov 02 14:43:03.386453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:03.386275 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" event={"ID":"c782b989-1334-4b2d-a7dd-1080b0951ea7","Type":"ContainerStarted","Data":"4c3a48471e41fed92d2afaf838e8f520e97f9b5ba07e3c11b67b176d56d6ae37"} Nov 02 14:43:03.441589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:03.441536 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" podStartSLOduration=1.441522073 podStartE2EDuration="1.441522073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:43:03.43887568 +0000 UTC m=+3532.164497454" watchObservedRunningTime="2026-11-02 14:43:03.441522073 +0000 UTC m=+3532.167143889" Nov 02 14:43:03.792454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:03.792422 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:03.975599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:03.973890 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:05.630435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:05.627719 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:07.487080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:07.487033 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:09.232264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:09.232227 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:11.048640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:11.048598 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:12.793422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:12.793385 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:12.803560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:12.803523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:12.829211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:12.829174 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:13.487310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:13.487280 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:43:14.129729 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:14.129696 5591 scope.go:122] "RemoveContainer" containerID="0638431cf0826a37d3423daccd4c6cd8bbe7e70330e3fa525f4994e5d1792c15" Nov 02 14:43:14.493005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:14.489411 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:16.512799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:16.512760 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:18.389009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:18.382823 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:20.169978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:20.169935 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:21.899641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:21.899600 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:23.602116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:23.597740 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:24.019329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:24.019298 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk"] Nov 02 14:43:24.019558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:24.019532 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" containerID="cri-o://3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042" gracePeriod=30 Nov 02 14:43:25.490006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:25.489616 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:27.054763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:27.054707 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:27.224127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:27.224096 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:27.389608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:27.389566 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:28.088308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:28.088271 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:43:28.088726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:28.088354 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:43:28.976005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:28.975098 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:29.315900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.315867 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:43:29.458574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.458535 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-envoy\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.458772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.458649 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1537b9-afe7-43be-81c0-0762bd50a35f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1537b9-afe7-43be-81c0-0762bd50a35f-istiod-ca-cert\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.458772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.458684 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-data\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.458772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.458746 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-credential-socket\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.458956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.458834 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-socket\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.459055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459028 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-token\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-token\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.459128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459110 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-credential-socket" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:29.459188 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459120 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-kube-api-access-np65x\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-kube-api-access-np65x\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.459258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459216 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1537b9-afe7-43be-81c0-0762bd50a35f-istiod-ca-cert" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:43:29.459258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459221 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-data" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:29.459258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459233 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-socket" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:29.459435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459268 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-certs\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.459435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459338 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-podinfo\") pod \"4c1537b9-afe7-43be-81c0-0762bd50a35f\" (UID: \"4c1537b9-afe7-43be-81c0-0762bd50a35f\") " Nov 02 14:43:29.459555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.459479 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-certs" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:29.460299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.460268 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.460432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.460302 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1537b9-afe7-43be-81c0-0762bd50a35f-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.460432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.460315 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.460432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.460328 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.460432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.460342 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.461308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.461280 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-envoy" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:29.461967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.461938 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-kube-api-access-np65x" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "kube-api-access-np65x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:29.462125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.462099 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-token" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:29.462281 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.462258 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-podinfo" pod "4c1537b9-afe7-43be-81c0-0762bd50a35f" (UID: "4c1537b9-afe7-43be-81c0-0762bd50a35f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:43:29.561554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.561459 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.561554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.561498 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.561554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.561507 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.561554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.561538 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np65x\" (UniqueName: \"kubernetes.io/projected/4c1537b9-afe7-43be-81c0-0762bd50a35f-kube-api-access-np65x\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:29.663421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.663378 5591 generic.go:356] "Generic (PLEG): container finished" podID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerID="3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042" exitCode=0 Nov 02 14:43:29.663614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.663464 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" Nov 02 14:43:29.663614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.663470 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" event={"ID":"4c1537b9-afe7-43be-81c0-0762bd50a35f","Type":"ContainerDied","Data":"3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042"} Nov 02 14:43:29.663614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.663561 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk" event={"ID":"4c1537b9-afe7-43be-81c0-0762bd50a35f","Type":"ContainerDied","Data":"f0f0d7e6031d5832fe3fd01b6734aa9adf1e491e12bf05de3fde60fbc8d711b8"} Nov 02 14:43:29.663614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.663589 5591 scope.go:122] "RemoveContainer" containerID="3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042" Nov 02 14:43:29.676569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.676549 5591 scope.go:122] "RemoveContainer" containerID="3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042" Nov 02 14:43:29.676849 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:43:29.676828 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042\": container with ID starting with 3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042 not found: ID does not exist" containerID="3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042" Nov 02 14:43:29.676893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.676860 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042"} err="failed to get container status \"3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042\": rpc error: code = NotFound desc = could not find container \"3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042\": container with ID starting with 3eb4b12e35860ee12c6e1e5a657dbb0a183002e78005edac886f310a2d8d4042 not found: ID does not exist" Nov 02 14:43:29.721561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.721530 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk"] Nov 02 14:43:29.734292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:29.734266 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-4svvk"] Nov 02 14:43:30.564007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.548494 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:30.935164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.935132 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf"] Nov 02 14:43:30.936097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.936074 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" Nov 02 14:43:30.936097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.936096 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" Nov 02 14:43:30.936365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.936349 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" containerName="istio-proxy" Nov 02 14:43:30.941807 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.941768 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:30.969096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.969018 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf"] Nov 02 14:43:30.976526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.976496 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-80133f\"/\"default-dockercfg-kqjt7\"" Nov 02 14:43:30.976682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.976655 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-80133f\"/\"kube-root-ca.crt\"" Nov 02 14:43:30.976753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.976738 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-80133f\"/\"openshift-service-ca.crt\"" Nov 02 14:43:30.976810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:30.976788 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-80133f\"/\"test-model-80133f-kserve-self-signed-certs\"" Nov 02 14:43:31.070376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.070342 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:31.076205 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:43:31.076175 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:43:31.089265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.089238 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-dshm\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.089400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.089280 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/79dea7ff-8473-4fb6-8325-294bcec1299c-kube-api-access-gbplv\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.089400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.089359 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79dea7ff-8473-4fb6-8325-294bcec1299c-tls-certs\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.089533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.089514 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-tmp-dir\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.089615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.089601 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-model-cache\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.089658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.089636 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-home\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.195057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.194671 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-dshm\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.195057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.194746 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/79dea7ff-8473-4fb6-8325-294bcec1299c-kube-api-access-gbplv\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.195057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.194791 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79dea7ff-8473-4fb6-8325-294bcec1299c-tls-certs\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.195057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.194953 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-tmp-dir\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.195431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.195084 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-model-cache\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.195431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.195140 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-home\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.200011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.195701 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-home\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.200011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.196862 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-tmp-dir\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.200011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.197351 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-model-cache\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.205697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.205668 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-dshm\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.236012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.230337 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79dea7ff-8473-4fb6-8325-294bcec1299c-tls-certs\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.236012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.231569 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/79dea7ff-8473-4fb6-8325-294bcec1299c-kube-api-access-gbplv\") pod \"test-model-80133f-kserve-848cdc4b77-lppkf\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.270392 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.270347 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:31.443928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.442738 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1537b9-afe7-43be-81c0-0762bd50a35f" path="/var/lib/kubelet/pods/4c1537b9-afe7-43be-81c0-0762bd50a35f/volumes" Nov 02 14:43:31.642680 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:43:31.642650 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79dea7ff_8473_4fb6_8325_294bcec1299c.slice/crio-a1b64cb715be4caaa7d3ff460d924fef22309ff3de392dc4966b1ef6f175a3f6 WatchSource:0}: Error finding container a1b64cb715be4caaa7d3ff460d924fef22309ff3de392dc4966b1ef6f175a3f6: Status 404 returned error can't find the container with id a1b64cb715be4caaa7d3ff460d924fef22309ff3de392dc4966b1ef6f175a3f6 Nov 02 14:43:31.643904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.643877 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf"] Nov 02 14:43:31.682549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:31.682519 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" event={"ID":"79dea7ff-8473-4fb6-8325-294bcec1299c","Type":"ContainerStarted","Data":"a1b64cb715be4caaa7d3ff460d924fef22309ff3de392dc4966b1ef6f175a3f6"} Nov 02 14:43:32.357308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:32.357243 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:32.693259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:32.693216 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" event={"ID":"79dea7ff-8473-4fb6-8325-294bcec1299c","Type":"ContainerStarted","Data":"21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b"} Nov 02 14:43:32.738200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:32.738143 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" podStartSLOduration=2.738129409 podStartE2EDuration="2.738129409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:43:32.73596017 +0000 UTC m=+3561.461581940" watchObservedRunningTime="2026-11-02 14:43:32.738129409 +0000 UTC m=+3561.463751173" Nov 02 14:43:33.271706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:33.271671 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:34.071006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:34.067110 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:35.596331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:35.596287 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:37.398006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:37.397051 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:39.087862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:39.087822 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:40.824692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:40.824158 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:41.271503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:41.271463 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:41.284420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:41.284382 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:41.434627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:41.434593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:41.844460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:41.844427 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:43:42.563223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:42.563179 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:43.044230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:43.044187 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw"] Nov 02 14:43:43.044751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:43.044394 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" containerID="cri-o://1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a" gracePeriod=30 Nov 02 14:43:44.051041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:44.051009 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:43:44.395061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:44.382105 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:46.136008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:46.129333 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:47.880035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:47.878577 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:48.140975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.140944 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Nov 02 14:43:48.141181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.141023 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Nov 02 14:43:48.275199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.275162 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7"] Nov 02 14:43:48.275437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.275411 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" containerID="cri-o://338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48" gracePeriod=30 Nov 02 14:43:48.358507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.358476 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:43:48.407309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407221 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-socket\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407333 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-envoy\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407441 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-credential-socket\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407471 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae3377c-0760-40ea-9259-6edc9e44d021-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae3377c-0760-40ea-9259-6edc9e44d021-istiod-ca-cert\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407525 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bae3377c-0760-40ea-9259-6edc9e44d021-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae3377c-0760-40ea-9259-6edc9e44d021-istio-podinfo\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407570 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-istio-token\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-istio-token\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407613 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-kube-api-access-tczxw\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-kube-api-access-tczxw\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407563 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-socket" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:48.407685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407654 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-data\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407683 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-credential-socket" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:48.407963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407706 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-certs\") pod \"bae3377c-0760-40ea-9259-6edc9e44d021\" (UID: \"bae3377c-0760-40ea-9259-6edc9e44d021\") " Nov 02 14:43:48.407963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407905 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-data" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:48.407963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407945 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae3377c-0760-40ea-9259-6edc9e44d021-istiod-ca-cert" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:43:48.408238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.407971 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-certs" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:48.408862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.408831 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.408862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.408865 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.409090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.408881 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae3377c-0760-40ea-9259-6edc9e44d021-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.409090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.408894 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.409090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.408906 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.410092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.410055 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-istio-token" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:48.410235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.410216 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-envoy" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:48.410349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.410324 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-kube-api-access-tczxw" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "kube-api-access-tczxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:48.410498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.410387 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bae3377c-0760-40ea-9259-6edc9e44d021-istio-podinfo" pod "bae3377c-0760-40ea-9259-6edc9e44d021" (UID: "bae3377c-0760-40ea-9259-6edc9e44d021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:43:48.511539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.511406 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae3377c-0760-40ea-9259-6edc9e44d021-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.511539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.511441 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.511539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.511458 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tczxw\" (UniqueName: \"kubernetes.io/projected/bae3377c-0760-40ea-9259-6edc9e44d021-kube-api-access-tczxw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.511539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.511472 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae3377c-0760-40ea-9259-6edc9e44d021-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:48.912215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.912164 5591 generic.go:356] "Generic (PLEG): container finished" podID="bae3377c-0760-40ea-9259-6edc9e44d021" containerID="1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a" exitCode=0 Nov 02 14:43:48.912215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.912211 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" Nov 02 14:43:48.912809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.912248 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" event={"ID":"bae3377c-0760-40ea-9259-6edc9e44d021","Type":"ContainerDied","Data":"1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a"} Nov 02 14:43:48.912809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.912326 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw" event={"ID":"bae3377c-0760-40ea-9259-6edc9e44d021","Type":"ContainerDied","Data":"33b242a8bfa0dc261b2e0e1e7197e0468fba73e9464919a7754a8c41b6f73199"} Nov 02 14:43:48.912809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.912357 5591 scope.go:122] "RemoveContainer" containerID="1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a" Nov 02 14:43:48.927491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.927467 5591 scope.go:122] "RemoveContainer" containerID="1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a" Nov 02 14:43:48.927795 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:43:48.927773 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a\": container with ID starting with 1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a not found: ID does not exist" containerID="1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a" Nov 02 14:43:48.927889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.927801 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a"} err="failed to get container status \"1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a\": rpc error: code = NotFound desc = could not find container \"1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a\": container with ID starting with 1aa6ae24eaf801823413da22d720609d2f2f0bc21133a47ef5e631cbd398d07a not found: ID does not exist" Nov 02 14:43:48.970020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.969978 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw"] Nov 02 14:43:48.982860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:48.982836 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-pg2qw"] Nov 02 14:43:49.435506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:49.435471 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" path="/var/lib/kubelet/pods/bae3377c-0760-40ea-9259-6edc9e44d021/volumes" Nov 02 14:43:49.728540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:49.728451 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:51.480008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:51.469931 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:51.587054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:51.587019 5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:43:51.587234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:51.587081 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:43:53.131468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.131423 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:53.566715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.566686 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:43:53.707330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707249 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-envoy\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.707330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707293 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-data\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.707330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707327 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-credential-socket\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.707645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707423 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-certs\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.707645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707535 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-podinfo\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.707645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707638 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-kube-api-access-xv7vz\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-kube-api-access-xv7vz\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.707818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707646 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-data" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:53.707818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707691 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f16efd-16c6-491a-91ff-2e42183de3fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f16efd-16c6-491a-91ff-2e42183de3fe-istiod-ca-cert\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.707818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707751 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-socket\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.707818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707772 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-credential-socket" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:53.708084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707852 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-token\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-token\") pod \"72f16efd-16c6-491a-91ff-2e42183de3fe\" (UID: \"72f16efd-16c6-491a-91ff-2e42183de3fe\") " Nov 02 14:43:53.708276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.707965 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-certs" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:53.708276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.708117 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-socket" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:53.708489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.708150 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f16efd-16c6-491a-91ff-2e42183de3fe-istiod-ca-cert" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:43:53.708613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.708597 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f16efd-16c6-491a-91ff-2e42183de3fe-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.708687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.708614 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.708687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.708623 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.708687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.708633 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.708687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.708641 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.709724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.709701 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-kube-api-access-xv7vz" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "kube-api-access-xv7vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:53.709865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.709842 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-podinfo" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:43:53.709953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.709931 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-envoy" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:43:53.710199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.710181 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-token" pod "72f16efd-16c6-491a-91ff-2e42183de3fe" (UID: "72f16efd-16c6-491a-91ff-2e42183de3fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:53.810694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.810657 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.810694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.810688 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.810694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.810701 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv7vz\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-kube-api-access-xv7vz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.810947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.810716 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f16efd-16c6-491a-91ff-2e42183de3fe-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:43:53.968646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.968555 5591 generic.go:356] "Generic (PLEG): container finished" podID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerID="338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48" exitCode=0 Nov 02 14:43:53.968646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.968590 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" event={"ID":"72f16efd-16c6-491a-91ff-2e42183de3fe","Type":"ContainerDied","Data":"338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48"} Nov 02 14:43:53.968646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.968623 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" Nov 02 14:43:53.968890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.968675 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7" event={"ID":"72f16efd-16c6-491a-91ff-2e42183de3fe","Type":"ContainerDied","Data":"5e1dadcf3180446b8a6a14dce6e41b8c1d0f2ab9d37b6c9b4933e47cdf659d75"} Nov 02 14:43:53.968890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.968698 5591 scope.go:122] "RemoveContainer" containerID="338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48" Nov 02 14:43:53.981295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.981274 5591 scope.go:122] "RemoveContainer" containerID="338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48" Nov 02 14:43:53.981606 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:43:53.981586 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48\": container with ID starting with 338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48 not found: ID does not exist" containerID="338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48" Nov 02 14:43:53.981687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:53.981613 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48"} err="failed to get container status \"338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48\": rpc error: code = NotFound desc = could not find container \"338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48\": container with ID starting with 338236c107e6d26a00fa75e6e423c05aaea4fd28e842ea250a2e04ded4c7dd48 not found: ID does not exist" Nov 02 14:43:54.027875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:54.027844 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7"] Nov 02 14:43:54.040259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:54.040230 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-npbx7"] Nov 02 14:43:54.858401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:54.858363 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:55.033319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:55.032560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:55.440961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:55.440916 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" path="/var/lib/kubelet/pods/72f16efd-16c6-491a-91ff-2e42183de3fe/volumes" Nov 02 14:43:56.573742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:56.572621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:43:58.375016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:58.374586 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:00.002258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:43:59.997888 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:01.075536 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:44:01.075492 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:44:01.675008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:01.672058 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:03.205006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:03.201195 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:04.794587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:04.794527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:06.360649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:06.360615 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:08.070971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:08.066643 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:09.756726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:09.756681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:11.550002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:11.536469 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:12.172094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:12.172067 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:44:12.226320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:12.226291 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:44:13.164067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:13.164029 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:14.807052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:14.807011 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:16.564617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:16.564564 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:17.719472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:17.717953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:18.496114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:18.495961 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:19.607007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:19.586161 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:19.981379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:19.981345 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:20.257115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:20.251595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:21.061091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:21.061056 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf"] Nov 02 14:44:21.061521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:21.061253 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" containerID="cri-o://575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf" gracePeriod=30 Nov 02 14:44:21.975717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:21.975657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:22.377582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:22.377486 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:44:22.378050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:22.377571 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:44:23.695031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:23.681306 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:25.401500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:25.401458 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:25.596755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:25.596712 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:25.759274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:25.759167 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:26.409536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.409506 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 14:44:26.455166 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.455133 5591 generic.go:356] "Generic (PLEG): container finished" podID="e739adc0-9202-4502-846d-089cb272b6db" containerID="575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf" exitCode=0 Nov 02 14:44:26.455356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.455180 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" Nov 02 14:44:26.455356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.455215 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" event={"ID":"e739adc0-9202-4502-846d-089cb272b6db","Type":"ContainerDied","Data":"575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf"} Nov 02 14:44:26.455356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.455267 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf" event={"ID":"e739adc0-9202-4502-846d-089cb272b6db","Type":"ContainerDied","Data":"04c18cfd25b8fdae77870fe62aed2331e33d1ec4fbe2dda56811f9ff65bdd4fb"} Nov 02 14:44:26.455356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.455291 5591 scope.go:122] "RemoveContainer" containerID="575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf" Nov 02 14:44:26.470733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.470705 5591 scope.go:122] "RemoveContainer" containerID="575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf" Nov 02 14:44:26.471790 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:44:26.471752 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf\": container with ID starting with 575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf not found: ID does not exist" containerID="575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf" Nov 02 14:44:26.471895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.471796 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf"} err="failed to get container status \"575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf\": rpc error: code = NotFound desc = could not find container \"575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf\": container with ID starting with 575c9e871562ec644c051da48211e268aecdea0b3fa11e474ce0bb2ebdcfb2cf not found: ID does not exist" Nov 02 14:44:26.584252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584216 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-data\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.584467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584392 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e739adc0-9202-4502-846d-089cb272b6db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e739adc0-9202-4502-846d-089cb272b6db-istiod-ca-cert\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.584467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584427 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-kube-api-access-gjdhr\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-kube-api-access-gjdhr\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.584587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584560 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e739adc0-9202-4502-846d-089cb272b6db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e739adc0-9202-4502-846d-089cb272b6db-istio-podinfo\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.584647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584583 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-data" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:26.584647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584608 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-credential-socket\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.584757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584654 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-istio-token\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-istio-token\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.584856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584833 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-certs\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.584933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584856 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e739adc0-9202-4502-846d-089cb272b6db-istiod-ca-cert" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:44:26.584933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.584924 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-envoy\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.585085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585041 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-credential-socket" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:26.585085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585053 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-socket\") pod \"e739adc0-9202-4502-846d-089cb272b6db\" (UID: \"e739adc0-9202-4502-846d-089cb272b6db\") " Nov 02 14:44:26.585244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585223 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-certs" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:26.585533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585506 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-socket" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:26.585931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585914 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.586078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585936 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.586078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585950 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.586078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585964 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.586078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.585979 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e739adc0-9202-4502-846d-089cb272b6db-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.587617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.587593 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e739adc0-9202-4502-846d-089cb272b6db-istio-podinfo" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:44:26.587866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.587841 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-istio-token" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:44:26.588030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.587961 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-envoy" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:26.588224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.588197 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-kube-api-access-gjdhr" pod "e739adc0-9202-4502-846d-089cb272b6db" (UID: "e739adc0-9202-4502-846d-089cb272b6db"). InnerVolumeSpecName "kube-api-access-gjdhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:44:26.611513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.611482 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:26.687843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.687762 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e739adc0-9202-4502-846d-089cb272b6db-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.687843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.687792 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.687843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.687801 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e739adc0-9202-4502-846d-089cb272b6db-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.687843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.687809 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdhr\" (UniqueName: \"kubernetes.io/projected/e739adc0-9202-4502-846d-089cb272b6db-kube-api-access-gjdhr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:26.805719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.805682 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf"] Nov 02 14:44:26.818737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:26.818707 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-gz8pf"] Nov 02 14:44:27.066060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:27.065081 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:27.236800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:27.236768 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:27.435177 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:27.435139 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e739adc0-9202-4502-846d-089cb272b6db" path="/var/lib/kubelet/pods/e739adc0-9202-4502-846d-089cb272b6db/volumes" Nov 02 14:44:28.744012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:28.738669 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:30.298357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:30.298281 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:31.076202 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:44:31.076162 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:44:31.966018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:31.963052 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:32.419641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:32.419604 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:33.558559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:33.558521 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:35.159646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:35.159600 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:35.833406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:35.833371 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:35.995621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:35.995589 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:36.892010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:36.887207 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:38.444471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:38.444375 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:40.264230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:40.264193 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:41.963349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:41.963310 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:43.031010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:43.029611 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2"] Nov 02 14:44:43.109563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:43.109473 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:43.726225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:43.726188 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:43.891234 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:43.891186 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:45.338685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:45.338646 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:47.067860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:47.067819 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:48.763150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:48.763118 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:48.795780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:48.795743 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n"] Nov 02 14:44:48.946657 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:48.946621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:49.115276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:49.115190 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:50.252917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:50.252884 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:50.586061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:50.581519 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:52.352021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:52.345245 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:53.796075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:53.796035 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" podUID="ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" containerName="payload-pre-processing" containerID="cri-o://0235655b7fa965f77c7fb07989a608883a5770a333ca4f2a0e4c5cf29860d27d" gracePeriod=25 Nov 02 14:44:54.083224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:54.082240 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:55.656005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:55.653892 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:56.215450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:56.215417 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:57.238005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:57.236854 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:58.055108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.055065 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" podUID="56d985b1-0165-471a-aad3-5dc60b3ce843" containerName="main" containerID="cri-o://527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd" gracePeriod=45 Nov 02 14:44:58.341237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.341208 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:44:58.409868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.409832 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-home\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-home\") pod \"56d985b1-0165-471a-aad3-5dc60b3ce843\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " Nov 02 14:44:58.410117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.410024 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d985b1-0165-471a-aad3-5dc60b3ce843-kube-api-access-r5l54\" (UniqueName: \"kubernetes.io/projected/56d985b1-0165-471a-aad3-5dc60b3ce843-kube-api-access-r5l54\") pod \"56d985b1-0165-471a-aad3-5dc60b3ce843\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " Nov 02 14:44:58.410217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.410120 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d985b1-0165-471a-aad3-5dc60b3ce843-tls-certs\" (UniqueName: \"kubernetes.io/secret/56d985b1-0165-471a-aad3-5dc60b3ce843-tls-certs\") pod \"56d985b1-0165-471a-aad3-5dc60b3ce843\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " Nov 02 14:44:58.410217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.410150 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-home" pod "56d985b1-0165-471a-aad3-5dc60b3ce843" (UID: "56d985b1-0165-471a-aad3-5dc60b3ce843"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:58.410339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.410214 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-dshm\") pod \"56d985b1-0165-471a-aad3-5dc60b3ce843\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " Nov 02 14:44:58.410339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.410255 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-tmp-dir\") pod \"56d985b1-0165-471a-aad3-5dc60b3ce843\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " Nov 02 14:44:58.410339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.410298 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-model-cache\") pod \"56d985b1-0165-471a-aad3-5dc60b3ce843\" (UID: \"56d985b1-0165-471a-aad3-5dc60b3ce843\") " Nov 02 14:44:58.410765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.410730 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-model-cache" pod "56d985b1-0165-471a-aad3-5dc60b3ce843" (UID: "56d985b1-0165-471a-aad3-5dc60b3ce843"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:58.411115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.411096 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-tmp-dir" pod "56d985b1-0165-471a-aad3-5dc60b3ce843" (UID: "56d985b1-0165-471a-aad3-5dc60b3ce843"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:58.412115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.412094 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:58.412238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.412226 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:58.412318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.412308 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:58.412694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.412674 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-dshm" pod "56d985b1-0165-471a-aad3-5dc60b3ce843" (UID: "56d985b1-0165-471a-aad3-5dc60b3ce843"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:44:58.413964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.413943 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d985b1-0165-471a-aad3-5dc60b3ce843-tls-certs" pod "56d985b1-0165-471a-aad3-5dc60b3ce843" (UID: "56d985b1-0165-471a-aad3-5dc60b3ce843"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:44:58.414053 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.414033 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d985b1-0165-471a-aad3-5dc60b3ce843-kube-api-access-r5l54" pod "56d985b1-0165-471a-aad3-5dc60b3ce843" (UID: "56d985b1-0165-471a-aad3-5dc60b3ce843"). InnerVolumeSpecName "kube-api-access-r5l54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:44:58.514013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.513961 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5l54\" (UniqueName: \"kubernetes.io/projected/56d985b1-0165-471a-aad3-5dc60b3ce843-kube-api-access-r5l54\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:58.514013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.514008 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d985b1-0165-471a-aad3-5dc60b3ce843-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:58.514013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.514018 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d985b1-0165-471a-aad3-5dc60b3ce843-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:44:58.743648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.743617 5591 generic.go:356] "Generic (PLEG): container finished" podID="56d985b1-0165-471a-aad3-5dc60b3ce843" containerID="527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd" exitCode=0 Nov 02 14:44:58.743837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.743659 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" Nov 02 14:44:58.743837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.743698 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" event={"ID":"56d985b1-0165-471a-aad3-5dc60b3ce843","Type":"ContainerDied","Data":"527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd"} Nov 02 14:44:58.743837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.743738 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2" event={"ID":"56d985b1-0165-471a-aad3-5dc60b3ce843","Type":"ContainerDied","Data":"69b9d344d9f0ea16954fa2835989fb0467875a21af8bad6a16ec723f56d48f56"} Nov 02 14:44:58.743837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.743756 5591 scope.go:122] "RemoveContainer" containerID="527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd" Nov 02 14:44:58.768699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.768677 5591 scope.go:122] "RemoveContainer" containerID="527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd" Nov 02 14:44:58.769030 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:44:58.769006 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd\": container with ID starting with 527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd not found: ID does not exist" containerID="527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd" Nov 02 14:44:58.769114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.769038 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd"} err="failed to get container status \"527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd\": rpc error: code = NotFound desc = could not find container \"527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd\": container with ID starting with 527c0d23f3065f56bf26dc7b48e5ce29e3bc38264e43dfd377952eddf4bad1fd not found: ID does not exist" Nov 02 14:44:58.801430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.801399 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2"] Nov 02 14:44:58.814051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.814011 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e1136e/test-model-e1136e-kserve-9854cbf77-ksrx2"] Nov 02 14:44:58.972126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:58.972089 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:44:59.436152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:44:59.436117 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d985b1-0165-471a-aad3-5dc60b3ce843" path="/var/lib/kubelet/pods/56d985b1-0165-471a-aad3-5dc60b3ce843/volumes" Nov 02 14:45:00.230942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.230910 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5"] Nov 02 14:45:00.232073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232045 5591 cpu_manager.go:414] "containerMap: removing container" podUID="56d985b1-0165-471a-aad3-5dc60b3ce843" containerName="main" Nov 02 14:45:00.232073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232070 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d985b1-0165-471a-aad3-5dc60b3ce843" containerName="main" Nov 02 14:45:00.232278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232089 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" Nov 02 14:45:00.232278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232097 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" Nov 02 14:45:00.232278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232159 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" Nov 02 14:45:00.232278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232166 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" Nov 02 14:45:00.232278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232205 5591 cpu_manager.go:414] "containerMap: removing container" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" Nov 02 14:45:00.232278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232212 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" Nov 02 14:45:00.232595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232424 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae3377c-0760-40ea-9259-6edc9e44d021" containerName="istio-proxy" Nov 02 14:45:00.232595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232442 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f16efd-16c6-491a-91ff-2e42183de3fe" containerName="istio-proxy" Nov 02 14:45:00.232595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232460 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d985b1-0165-471a-aad3-5dc60b3ce843" containerName="main" Nov 02 14:45:00.232595 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.232473 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e739adc0-9202-4502-846d-089cb272b6db" containerName="istio-proxy" Nov 02 14:45:00.245745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.245661 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl"] Nov 02 14:45:00.250571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.250022 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:00.255671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.254417 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn"] Nov 02 14:45:00.255671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.254503 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:00.263706 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.263678 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5"] Nov 02 14:45:00.263843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.263728 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p"] Nov 02 14:45:00.273670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.269019 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:00.273670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.269646 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn"] Nov 02 14:45:00.273670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.269735 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:00.284273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.277838 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6"] Nov 02 14:45:00.289480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.288802 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl"] Nov 02 14:45:00.289480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.289030 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\"" Nov 02 14:45:00.293453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.292089 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:00.296864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.296835 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg"] Nov 02 14:45:00.326358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.324735 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6"] Nov 02 14:45:00.326358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.324810 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv"] Nov 02 14:45:00.328564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.328533 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.341915 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl6pd\" (UniqueName: \"kubernetes.io/projected/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-kube-api-access-fl6pd\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p\" (UID: \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342061 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrdn\" (UniqueName: \"kubernetes.io/projected/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-kube-api-access-wcrdn\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn\" (UID: \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342097 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksjcw\" (UniqueName: \"kubernetes.io/projected/d6982912-1475-49a1-a605-2db8b5bc7687-kube-api-access-ksjcw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5\" (UID: \"d6982912-1475-49a1-a605-2db8b5bc7687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342180 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f48a2c-7440-4363-bfc0-a7ccafdbed80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl\" (UID: \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342217 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62221ac5-75fe-4999-9df5-2952cd3dee73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg\" (UID: \"62221ac5-75fe-4999-9df5-2952cd3dee73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342291 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p\" (UID: \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342326 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-956pg\" (UniqueName: \"kubernetes.io/projected/d0947e44-e37f-4591-adfe-714233cdbd98-kube-api-access-956pg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6\" (UID: \"d0947e44-e37f-4591-adfe-714233cdbd98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342385 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vwz\" (UniqueName: \"kubernetes.io/projected/70f48a2c-7440-4363-bfc0-a7ccafdbed80-kube-api-access-m8vwz\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl\" (UID: \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342436 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0947e44-e37f-4591-adfe-714233cdbd98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6\" (UID: \"d0947e44-e37f-4591-adfe-714233cdbd98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342465 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6982912-1475-49a1-a605-2db8b5bc7687-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5\" (UID: \"d6982912-1475-49a1-a605-2db8b5bc7687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342552 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn\" (UID: \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.342591 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9t6w\" (UniqueName: \"kubernetes.io/projected/62221ac5-75fe-4999-9df5-2952cd3dee73-kube-api-access-g9t6w\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg\" (UID: \"62221ac5-75fe-4999-9df5-2952cd3dee73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:00.346579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.345199 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l"] Nov 02 14:45:00.357827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.355828 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:00.376090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.374562 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p"] Nov 02 14:45:00.376090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.374641 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv"] Nov 02 14:45:00.376090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.374673 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc"] Nov 02 14:45:00.387218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.382306 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:00.444114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444066 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7a7fb7-de79-44d6-ac70-753b949d5a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv\" (UID: \"df7a7fb7-de79-44d6-ac70-753b949d5a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:00.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444158 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce129d-05f8-48db-82cb-fa836c2992a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l\" (UID: \"b9ce129d-05f8-48db-82cb-fa836c2992a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:00.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444192 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbx7j\" (UniqueName: \"kubernetes.io/projected/df7a7fb7-de79-44d6-ac70-753b949d5a12-kube-api-access-gbx7j\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv\" (UID: \"df7a7fb7-de79-44d6-ac70-753b949d5a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:00.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444248 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrdn\" (UniqueName: \"kubernetes.io/projected/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-kube-api-access-wcrdn\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn\" (UID: \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:00.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444276 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksjcw\" (UniqueName: \"kubernetes.io/projected/d6982912-1475-49a1-a605-2db8b5bc7687-kube-api-access-ksjcw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5\" (UID: \"d6982912-1475-49a1-a605-2db8b5bc7687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:00.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444364 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f48a2c-7440-4363-bfc0-a7ccafdbed80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl\" (UID: \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:00.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444407 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62221ac5-75fe-4999-9df5-2952cd3dee73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg\" (UID: \"62221ac5-75fe-4999-9df5-2952cd3dee73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:00.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444487 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p\" (UID: \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:00.444556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444517 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-956pg\" (UniqueName: \"kubernetes.io/projected/d0947e44-e37f-4591-adfe-714233cdbd98-kube-api-access-956pg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6\" (UID: \"d0947e44-e37f-4591-adfe-714233cdbd98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:00.445078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444770 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg"] Nov 02 14:45:00.445078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444808 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l"] Nov 02 14:45:00.445078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.444836 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42"] Nov 02 14:45:00.445883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.445855 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:00.446736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.446708 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vwz\" (UniqueName: \"kubernetes.io/projected/70f48a2c-7440-4363-bfc0-a7ccafdbed80-kube-api-access-m8vwz\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl\" (UID: \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:00.446848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.446775 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v68kn\" (UniqueName: \"kubernetes.io/projected/b9ce129d-05f8-48db-82cb-fa836c2992a0-kube-api-access-v68kn\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l\" (UID: \"b9ce129d-05f8-48db-82cb-fa836c2992a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:00.447227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.446812 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0947e44-e37f-4591-adfe-714233cdbd98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6\" (UID: \"d0947e44-e37f-4591-adfe-714233cdbd98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:00.447336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.447257 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6982912-1475-49a1-a605-2db8b5bc7687-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5\" (UID: \"d6982912-1475-49a1-a605-2db8b5bc7687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:00.447414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.447349 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn\" (UID: \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:00.447414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.447397 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9t6w\" (UniqueName: \"kubernetes.io/projected/62221ac5-75fe-4999-9df5-2952cd3dee73-kube-api-access-g9t6w\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg\" (UID: \"62221ac5-75fe-4999-9df5-2952cd3dee73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:00.447543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.447465 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl6pd\" (UniqueName: \"kubernetes.io/projected/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-kube-api-access-fl6pd\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p\" (UID: \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:00.448396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.448370 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0947e44-e37f-4591-adfe-714233cdbd98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6\" (UID: \"d0947e44-e37f-4591-adfe-714233cdbd98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:00.448542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.448515 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6982912-1475-49a1-a605-2db8b5bc7687-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5\" (UID: \"d6982912-1475-49a1-a605-2db8b5bc7687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:00.449352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.449324 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn\" (UID: \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:00.449528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.449505 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p\" (UID: \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:00.449909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.449824 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f48a2c-7440-4363-bfc0-a7ccafdbed80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl\" (UID: \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:00.450091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.450067 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62221ac5-75fe-4999-9df5-2952cd3dee73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg\" (UID: \"62221ac5-75fe-4999-9df5-2952cd3dee73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:00.452277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.452253 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6"] Nov 02 14:45:00.460252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.454245 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:00.460252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.458787 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc"] Nov 02 14:45:00.460252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.458829 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6"] Nov 02 14:45:00.460252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.458851 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv"] Nov 02 14:45:00.460252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.459188 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:00.464867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.464846 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42"] Nov 02 14:45:00.465819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.465755 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw"] Nov 02 14:45:00.466157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.466137 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:00.471629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.471609 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv"] Nov 02 14:45:00.471728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.471709 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:00.475001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.474967 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw"] Nov 02 14:45:00.475974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.475949 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-956pg\" (UniqueName: \"kubernetes.io/projected/d0947e44-e37f-4591-adfe-714233cdbd98-kube-api-access-956pg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6\" (UID: \"d0947e44-e37f-4591-adfe-714233cdbd98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:00.491310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.491249 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl6pd\" (UniqueName: \"kubernetes.io/projected/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-kube-api-access-fl6pd\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p\" (UID: \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:00.493224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.493197 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vwz\" (UniqueName: \"kubernetes.io/projected/70f48a2c-7440-4363-bfc0-a7ccafdbed80-kube-api-access-m8vwz\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl\" (UID: \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:00.494241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.494216 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrdn\" (UniqueName: \"kubernetes.io/projected/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-kube-api-access-wcrdn\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn\" (UID: \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:00.494376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.494354 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9t6w\" (UniqueName: \"kubernetes.io/projected/62221ac5-75fe-4999-9df5-2952cd3dee73-kube-api-access-g9t6w\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg\" (UID: \"62221ac5-75fe-4999-9df5-2952cd3dee73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:00.496066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.496040 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksjcw\" (UniqueName: \"kubernetes.io/projected/d6982912-1475-49a1-a605-2db8b5bc7687-kube-api-access-ksjcw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5\" (UID: \"d6982912-1475-49a1-a605-2db8b5bc7687\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:00.548736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.548692 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7a7fb7-de79-44d6-ac70-753b949d5a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv\" (UID: \"df7a7fb7-de79-44d6-ac70-753b949d5a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:00.548893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.548810 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce129d-05f8-48db-82cb-fa836c2992a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l\" (UID: \"b9ce129d-05f8-48db-82cb-fa836c2992a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:00.548893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.548840 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbx7j\" (UniqueName: \"kubernetes.io/projected/df7a7fb7-de79-44d6-ac70-753b949d5a12-kube-api-access-gbx7j\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv\" (UID: \"df7a7fb7-de79-44d6-ac70-753b949d5a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:00.548893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.548886 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96800fad-b801-4f01-91db-db678d597bf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv\" (UID: \"96800fad-b801-4f01-91db-db678d597bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:00.549125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.548911 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw\" (UID: \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:00.549125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549065 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgjq7\" (UniqueName: \"kubernetes.io/projected/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-kube-api-access-kgjq7\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6\" (UID: \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:00.549125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549088 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw67f\" (UniqueName: \"kubernetes.io/projected/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-kube-api-access-pw67f\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc\" (UID: \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:00.549125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549117 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc\" (UID: \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:00.549377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549134 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd174ecd-1998-4f54-8588-9de900552feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42\" (UID: \"cd174ecd-1998-4f54-8588-9de900552feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:00.549377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549172 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6\" (UID: \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:00.549377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549224 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v68kn\" (UniqueName: \"kubernetes.io/projected/b9ce129d-05f8-48db-82cb-fa836c2992a0-kube-api-access-v68kn\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l\" (UID: \"b9ce129d-05f8-48db-82cb-fa836c2992a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:00.549377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549309 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkdb\" (UniqueName: \"kubernetes.io/projected/cd174ecd-1998-4f54-8588-9de900552feb-kube-api-access-tpkdb\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42\" (UID: \"cd174ecd-1998-4f54-8588-9de900552feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:00.549377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549364 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7gm\" (UniqueName: \"kubernetes.io/projected/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-kube-api-access-gs7gm\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw\" (UID: \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:00.549665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549401 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dp6\" (UniqueName: \"kubernetes.io/projected/96800fad-b801-4f01-91db-db678d597bf1-kube-api-access-89dp6\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv\" (UID: \"96800fad-b801-4f01-91db-db678d597bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:00.549665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.549516 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7a7fb7-de79-44d6-ac70-753b949d5a12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv\" (UID: \"df7a7fb7-de79-44d6-ac70-753b949d5a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:00.550324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.550300 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce129d-05f8-48db-82cb-fa836c2992a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l\" (UID: \"b9ce129d-05f8-48db-82cb-fa836c2992a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:00.576403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.576371 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v68kn\" (UniqueName: \"kubernetes.io/projected/b9ce129d-05f8-48db-82cb-fa836c2992a0-kube-api-access-v68kn\") pod \"maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l\" (UID: \"b9ce129d-05f8-48db-82cb-fa836c2992a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:00.576584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.576560 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbx7j\" (UniqueName: \"kubernetes.io/projected/df7a7fb7-de79-44d6-ac70-753b949d5a12-kube-api-access-gbx7j\") pod \"maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv\" (UID: \"df7a7fb7-de79-44d6-ac70-753b949d5a12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:00.592697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.592674 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:00.641848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.641811 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.651515 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgjq7\" (UniqueName: \"kubernetes.io/projected/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-kube-api-access-kgjq7\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6\" (UID: \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.651575 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw67f\" (UniqueName: \"kubernetes.io/projected/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-kube-api-access-pw67f\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc\" (UID: \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.651614 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc\" (UID: \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.651641 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd174ecd-1998-4f54-8588-9de900552feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42\" (UID: \"cd174ecd-1998-4f54-8588-9de900552feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.651687 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6\" (UID: \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.651837 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkdb\" (UniqueName: \"kubernetes.io/projected/cd174ecd-1998-4f54-8588-9de900552feb-kube-api-access-tpkdb\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42\" (UID: \"cd174ecd-1998-4f54-8588-9de900552feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.651894 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7gm\" (UniqueName: \"kubernetes.io/projected/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-kube-api-access-gs7gm\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw\" (UID: \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.651919 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dp6\" (UniqueName: \"kubernetes.io/projected/96800fad-b801-4f01-91db-db678d597bf1-kube-api-access-89dp6\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv\" (UID: \"96800fad-b801-4f01-91db-db678d597bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.652056 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96800fad-b801-4f01-91db-db678d597bf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv\" (UID: \"96800fad-b801-4f01-91db-db678d597bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:00.652777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.652085 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw\" (UID: \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:00.655759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.653058 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw\" (UID: \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:00.655759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.653804 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6\" (UID: \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:00.655759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.654349 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96800fad-b801-4f01-91db-db678d597bf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv\" (UID: \"96800fad-b801-4f01-91db-db678d597bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:00.656545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.656519 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc\" (UID: \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:00.657382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.657359 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd174ecd-1998-4f54-8588-9de900552feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42\" (UID: \"cd174ecd-1998-4f54-8588-9de900552feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:00.686589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.680558 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:00.750419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.718920 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:00.750419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.722148 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:00.750419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.722917 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkdb\" (UniqueName: \"kubernetes.io/projected/cd174ecd-1998-4f54-8588-9de900552feb-kube-api-access-tpkdb\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42\" (UID: \"cd174ecd-1998-4f54-8588-9de900552feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:00.750419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.726568 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dp6\" (UniqueName: \"kubernetes.io/projected/96800fad-b801-4f01-91db-db678d597bf1-kube-api-access-89dp6\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv\" (UID: \"96800fad-b801-4f01-91db-db678d597bf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:00.750419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.730908 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7gm\" (UniqueName: \"kubernetes.io/projected/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-kube-api-access-gs7gm\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw\" (UID: \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:00.750419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.745279 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:00.763722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.763478 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:00.774492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.774282 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:00.776255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.776183 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgjq7\" (UniqueName: \"kubernetes.io/projected/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-kube-api-access-kgjq7\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6\" (UID: \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:00.792506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.792441 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw67f\" (UniqueName: \"kubernetes.io/projected/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-kube-api-access-pw67f\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc\" (UID: \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:00.827472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.827145 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:00.831355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.831287 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:00.859182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.850475 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:00.866353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.863376 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:00.866353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.863489 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:00.870246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:00.870184 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:01.120664 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:01.120575 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:45:01.132552 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.132517 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6982912_1475_49a1_a605_2db8b5bc7687.slice/crio-8ad1695987ff744f3fba657bb77a1f1e637e4c9cb62080e0b92e10c47b6aafda WatchSource:0}: Error finding container 8ad1695987ff744f3fba657bb77a1f1e637e4c9cb62080e0b92e10c47b6aafda: Status 404 returned error can't find the container with id 8ad1695987ff744f3fba657bb77a1f1e637e4c9cb62080e0b92e10c47b6aafda Nov 02 14:45:01.136936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.136703 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5"] Nov 02 14:45:01.275306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.275275 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl"] Nov 02 14:45:01.313386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.313181 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p"] Nov 02 14:45:01.320411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.320388 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn"] Nov 02 14:45:01.325773 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.325748 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6"] Nov 02 14:45:01.337025 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.336708 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f48a2c_7440_4363_bfc0_a7ccafdbed80.slice/crio-15952698a75779c795f4e9db20a7af54b6d03bcfffa4cefa00e54bdaec7b1ae4 WatchSource:0}: Error finding container 15952698a75779c795f4e9db20a7af54b6d03bcfffa4cefa00e54bdaec7b1ae4: Status 404 returned error can't find the container with id 15952698a75779c795f4e9db20a7af54b6d03bcfffa4cefa00e54bdaec7b1ae4 Nov 02 14:45:01.341125 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.341095 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59d93eb_2fa4_4094_98df_b0f1d75e0c94.slice/crio-fc9895329272f424b53f39965c83dad090605ab51c3ff19006ac2b8b39b11cbc WatchSource:0}: Error finding container fc9895329272f424b53f39965c83dad090605ab51c3ff19006ac2b8b39b11cbc: Status 404 returned error can't find the container with id fc9895329272f424b53f39965c83dad090605ab51c3ff19006ac2b8b39b11cbc Nov 02 14:45:01.341867 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.341842 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0947e44_e37f_4591_adfe_714233cdbd98.slice/crio-26677b5e29acc9e4458b271f7a8af54e90be7c044535611d90bace010c913289 WatchSource:0}: Error finding container 26677b5e29acc9e4458b271f7a8af54e90be7c044535611d90bace010c913289: Status 404 returned error can't find the container with id 26677b5e29acc9e4458b271f7a8af54e90be7c044535611d90bace010c913289 Nov 02 14:45:01.389722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.389477 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg"] Nov 02 14:45:01.415701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.415672 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv"] Nov 02 14:45:01.474284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.474251 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc"] Nov 02 14:45:01.478921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.478889 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l"] Nov 02 14:45:01.514046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.514012 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42"] Nov 02 14:45:01.541358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.541323 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw"] Nov 02 14:45:01.546577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.546555 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6"] Nov 02 14:45:01.579690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.579665 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv"] Nov 02 14:45:01.637696 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.637637 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91775710_4a1d_4dbb_ab4d_d3c771f4cad2.slice/crio-b2875f0ea15901e47f60e6ab897d51753ff78df694c419d4fb8632d849f1cf79 WatchSource:0}: Error finding container b2875f0ea15901e47f60e6ab897d51753ff78df694c419d4fb8632d849f1cf79: Status 404 returned error can't find the container with id b2875f0ea15901e47f60e6ab897d51753ff78df694c419d4fb8632d849f1cf79 Nov 02 14:45:01.638897 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.638758 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ce129d_05f8_48db_82cb_fa836c2992a0.slice/crio-0ad74f3059783bcf11cb8613d6684808bf0e953bd0963d18d5844a8ad719b051 WatchSource:0}: Error finding container 0ad74f3059783bcf11cb8613d6684808bf0e953bd0963d18d5844a8ad719b051: Status 404 returned error can't find the container with id 0ad74f3059783bcf11cb8613d6684808bf0e953bd0963d18d5844a8ad719b051 Nov 02 14:45:01.639917 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.639893 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd174ecd_1998_4f54_8588_9de900552feb.slice/crio-a5addec6796896aca71c5978c80ee87f6bb0d075469772a55031f66bbcc44203 WatchSource:0}: Error finding container a5addec6796896aca71c5978c80ee87f6bb0d075469772a55031f66bbcc44203: Status 404 returned error can't find the container with id a5addec6796896aca71c5978c80ee87f6bb0d075469772a55031f66bbcc44203 Nov 02 14:45:01.641430 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.641406 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d03007_f6a2_47d4_8742_4ca9d4d2f71c.slice/crio-18efc21e32b8bd221750bee00a59a3d77ad49dcf879adafb70680f3f7bcf6b75 WatchSource:0}: Error finding container 18efc21e32b8bd221750bee00a59a3d77ad49dcf879adafb70680f3f7bcf6b75: Status 404 returned error can't find the container with id 18efc21e32b8bd221750bee00a59a3d77ad49dcf879adafb70680f3f7bcf6b75 Nov 02 14:45:01.642691 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.642670 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f73d79_17ec_4ba0_a85c_08ba05fd9af6.slice/crio-c348a4417912c0b0ebccfb02a92296bd8a909c769673392f35a25e87b40895b5 WatchSource:0}: Error finding container c348a4417912c0b0ebccfb02a92296bd8a909c769673392f35a25e87b40895b5: Status 404 returned error can't find the container with id c348a4417912c0b0ebccfb02a92296bd8a909c769673392f35a25e87b40895b5 Nov 02 14:45:01.644248 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:45:01.644219 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96800fad_b801_4f01_91db_db678d597bf1.slice/crio-ac47c4837f3f02b4d4fa29fd689160f3609749cae17c41f54e2bff49cb455311 WatchSource:0}: Error finding container ac47c4837f3f02b4d4fa29fd689160f3609749cae17c41f54e2bff49cb455311: Status 404 returned error can't find the container with id ac47c4837f3f02b4d4fa29fd689160f3609749cae17c41f54e2bff49cb455311 Nov 02 14:45:01.775941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.775906 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" event={"ID":"e59d93eb-2fa4-4094-98df-b0f1d75e0c94","Type":"ContainerStarted","Data":"fc9895329272f424b53f39965c83dad090605ab51c3ff19006ac2b8b39b11cbc"} Nov 02 14:45:01.777590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.777561 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" event={"ID":"70f48a2c-7440-4363-bfc0-a7ccafdbed80","Type":"ContainerStarted","Data":"15952698a75779c795f4e9db20a7af54b6d03bcfffa4cefa00e54bdaec7b1ae4"} Nov 02 14:45:01.779126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.779102 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" event={"ID":"df7a7fb7-de79-44d6-ac70-753b949d5a12","Type":"ContainerStarted","Data":"fa7db9d2d94fd90b3dfdb5db588b4eec23835d39542eaba2f12857857cf0efaa"} Nov 02 14:45:01.780951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.780926 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" event={"ID":"85f73d79-17ec-4ba0-a85c-08ba05fd9af6","Type":"ContainerStarted","Data":"c348a4417912c0b0ebccfb02a92296bd8a909c769673392f35a25e87b40895b5"} Nov 02 14:45:01.782755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.782731 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" event={"ID":"96800fad-b801-4f01-91db-db678d597bf1","Type":"ContainerStarted","Data":"ac47c4837f3f02b4d4fa29fd689160f3609749cae17c41f54e2bff49cb455311"} Nov 02 14:45:01.784354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.784320 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" event={"ID":"62221ac5-75fe-4999-9df5-2952cd3dee73","Type":"ContainerStarted","Data":"f34d4faa1ca71a46a67cc3257433d7594f06038a3c1cafd995b363b6b3143539"} Nov 02 14:45:01.786474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.786447 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" event={"ID":"91775710-4a1d-4dbb-ab4d-d3c771f4cad2","Type":"ContainerStarted","Data":"b2875f0ea15901e47f60e6ab897d51753ff78df694c419d4fb8632d849f1cf79"} Nov 02 14:45:01.788180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.788106 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" event={"ID":"b9ce129d-05f8-48db-82cb-fa836c2992a0","Type":"ContainerStarted","Data":"0ad74f3059783bcf11cb8613d6684808bf0e953bd0963d18d5844a8ad719b051"} Nov 02 14:45:01.789789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.789762 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" event={"ID":"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c","Type":"ContainerStarted","Data":"18efc21e32b8bd221750bee00a59a3d77ad49dcf879adafb70680f3f7bcf6b75"} Nov 02 14:45:01.791130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.791107 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" event={"ID":"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8","Type":"ContainerStarted","Data":"f23f8c94078f4e9747c5f872fc6c93326cd20a6954c70cfe08b3b9fe9554647b"} Nov 02 14:45:01.792313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.792290 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" event={"ID":"d6982912-1475-49a1-a605-2db8b5bc7687","Type":"ContainerStarted","Data":"8ad1695987ff744f3fba657bb77a1f1e637e4c9cb62080e0b92e10c47b6aafda"} Nov 02 14:45:01.793429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.793409 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" event={"ID":"d0947e44-e37f-4591-adfe-714233cdbd98","Type":"ContainerStarted","Data":"26677b5e29acc9e4458b271f7a8af54e90be7c044535611d90bace010c913289"} Nov 02 14:45:01.794564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:01.794544 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" event={"ID":"cd174ecd-1998-4f54-8588-9de900552feb","Type":"ContainerStarted","Data":"a5addec6796896aca71c5978c80ee87f6bb0d075469772a55031f66bbcc44203"} Nov 02 14:45:02.363751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.363662 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:02.805458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.805415 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" event={"ID":"e59d93eb-2fa4-4094-98df-b0f1d75e0c94","Type":"ContainerStarted","Data":"3596f01a759a4e679a481bc04e3d394b15afdbdc58177540782fe8c80409e182"} Nov 02 14:45:02.807397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.807361 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" event={"ID":"70f48a2c-7440-4363-bfc0-a7ccafdbed80","Type":"ContainerStarted","Data":"87e5f68e4440c3ade2b39821678eb882389080bf9e8defcd55c3ee95cc69f3f4"} Nov 02 14:45:02.808968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.808938 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" event={"ID":"df7a7fb7-de79-44d6-ac70-753b949d5a12","Type":"ContainerStarted","Data":"6576f3a061960edea2951a380ffb7efb1a655de728d2dd1e0b9513fc6edb073d"} Nov 02 14:45:02.810507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.810483 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" event={"ID":"85f73d79-17ec-4ba0-a85c-08ba05fd9af6","Type":"ContainerStarted","Data":"7ee5a1fa0005d3db5365b6b5448bd53690e26e0dfadf6270bc948436a5ab4f22"} Nov 02 14:45:02.812153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.812132 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" event={"ID":"96800fad-b801-4f01-91db-db678d597bf1","Type":"ContainerStarted","Data":"16a3f7b59015cddd69da0bb07f6bee054c1e541f38de14f50a11719c8039cd0b"} Nov 02 14:45:02.813640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.813614 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" event={"ID":"62221ac5-75fe-4999-9df5-2952cd3dee73","Type":"ContainerStarted","Data":"3b70c657ae37f9392a2893f6329fa6b9c6e733a8224c1b7bf8b142df3de9f0ac"} Nov 02 14:45:02.815135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.815110 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" event={"ID":"91775710-4a1d-4dbb-ab4d-d3c771f4cad2","Type":"ContainerStarted","Data":"15707fd1106a80f430d33f5a94ac9dddfb546c395d0f33c8e056b049b4d20133"} Nov 02 14:45:02.821897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.821875 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" event={"ID":"b9ce129d-05f8-48db-82cb-fa836c2992a0","Type":"ContainerStarted","Data":"78f43a89fc7f2958f5fa603708870df6dcec8b8d84716606fc3739d307e8d4be"} Nov 02 14:45:02.823549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.823526 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" event={"ID":"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c","Type":"ContainerStarted","Data":"30b52cbbef483b08ee11b76eb5e36268a8d1e681a057f8a3c5d315959dbb42c5"} Nov 02 14:45:02.825145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.825121 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" event={"ID":"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8","Type":"ContainerStarted","Data":"894f16ad061f9f6d1d3f0a54c81fd15ed40c9f63a2485197fb9b36dfa5f4a36e"} Nov 02 14:45:02.826628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.826603 5591 generic.go:356] "Generic (PLEG): container finished" podID="d6982912-1475-49a1-a605-2db8b5bc7687" containerID="3aa06552bb6980e808c3aa571ee4bcb6f7866d615f95a4edb729e6669f6ac1aa" exitCode=60 Nov 02 14:45:02.826714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.826678 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" event={"ID":"d6982912-1475-49a1-a605-2db8b5bc7687","Type":"ContainerDied","Data":"3aa06552bb6980e808c3aa571ee4bcb6f7866d615f95a4edb729e6669f6ac1aa"} Nov 02 14:45:02.826883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.826857 5591 scope.go:122] "RemoveContainer" containerID="3aa06552bb6980e808c3aa571ee4bcb6f7866d615f95a4edb729e6669f6ac1aa" Nov 02 14:45:02.828432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.828408 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" event={"ID":"d0947e44-e37f-4591-adfe-714233cdbd98","Type":"ContainerStarted","Data":"9ce4a10c142c132d70d08eb6f02262772c2345d5495f44391da6df11101457b0"} Nov 02 14:45:02.830093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.830071 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" event={"ID":"cd174ecd-1998-4f54-8588-9de900552feb","Type":"ContainerStarted","Data":"02e1d759e376302c49fe32257feb0df18c35a52cc4cbffa2147d796d40d7bb5d"} Nov 02 14:45:02.897757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.897704 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" podStartSLOduration=2.89768858 podStartE2EDuration="2.89768858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:02.854708816 +0000 UTC m=+3651.580330587" watchObservedRunningTime="2026-11-02 14:45:02.89768858 +0000 UTC m=+3651.623310393" Nov 02 14:45:02.940757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.940697 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" podStartSLOduration=2.94067709 podStartE2EDuration="2.94067709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:02.938488776 +0000 UTC m=+3651.664110551" watchObservedRunningTime="2026-11-02 14:45:02.94067709 +0000 UTC m=+3651.666298875" Nov 02 14:45:02.983107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:02.983053 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" podStartSLOduration=2.9830389630000003 podStartE2EDuration="2.983038963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:02.979800535 +0000 UTC m=+3651.705422304" watchObservedRunningTime="2026-11-02 14:45:02.983038963 +0000 UTC m=+3651.708660733" Nov 02 14:45:03.021408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.021296 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" podStartSLOduration=3.021279063 podStartE2EDuration="3.021279063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.021082374 +0000 UTC m=+3651.746704144" watchObservedRunningTime="2026-11-02 14:45:03.021279063 +0000 UTC m=+3651.746900834" Nov 02 14:45:03.063323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.063272 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" podStartSLOduration=3.063258136 podStartE2EDuration="3.063258136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.061428917 +0000 UTC m=+3651.787050687" watchObservedRunningTime="2026-11-02 14:45:03.063258136 +0000 UTC m=+3651.788879905" Nov 02 14:45:03.104889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.104832 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" podStartSLOduration=3.104816961 podStartE2EDuration="3.104816961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.10275492 +0000 UTC m=+3651.828376693" watchObservedRunningTime="2026-11-02 14:45:03.104816961 +0000 UTC m=+3651.830438730" Nov 02 14:45:03.145157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.145098 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" podStartSLOduration=3.145077641 podStartE2EDuration="3.145077641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.143450467 +0000 UTC m=+3651.869072238" watchObservedRunningTime="2026-11-02 14:45:03.145077641 +0000 UTC m=+3651.870699411" Nov 02 14:45:03.185130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.185082 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" podStartSLOduration=3.185069231 podStartE2EDuration="3.185069231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.184157513 +0000 UTC m=+3651.909779284" watchObservedRunningTime="2026-11-02 14:45:03.185069231 +0000 UTC m=+3651.910691001" Nov 02 14:45:03.201050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.200962 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" podUID="ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:45:03.228388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.228326 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" podStartSLOduration=3.228310638 podStartE2EDuration="3.228310638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.227213455 +0000 UTC m=+3651.952835225" watchObservedRunningTime="2026-11-02 14:45:03.228310638 +0000 UTC m=+3651.953932408" Nov 02 14:45:03.271646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.271535 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" podStartSLOduration=3.271523266 podStartE2EDuration="3.271523266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.26888798 +0000 UTC m=+3651.994509750" watchObservedRunningTime="2026-11-02 14:45:03.271523266 +0000 UTC m=+3651.997145079" Nov 02 14:45:03.312656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.312604 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" podStartSLOduration=3.312592054 podStartE2EDuration="3.312592054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.311414188 +0000 UTC m=+3652.037035951" watchObservedRunningTime="2026-11-02 14:45:03.312592054 +0000 UTC m=+3652.038213824" Nov 02 14:45:03.354191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.354132 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" podStartSLOduration=3.354119564 podStartE2EDuration="3.354119564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.352474464 +0000 UTC m=+3652.078096235" watchObservedRunningTime="2026-11-02 14:45:03.354119564 +0000 UTC m=+3652.079741348" Nov 02 14:45:03.841020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.840910 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" event={"ID":"d6982912-1475-49a1-a605-2db8b5bc7687","Type":"ContainerStarted","Data":"fbc97338b4e8cef3445b8018071bc430421dc76e9c5a785efdefd03d9ae2a8f1"} Nov 02 14:45:03.842583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.842555 5591 generic.go:356] "Generic (PLEG): container finished" podID="d0947e44-e37f-4591-adfe-714233cdbd98" containerID="9ce4a10c142c132d70d08eb6f02262772c2345d5495f44391da6df11101457b0" exitCode=60 Nov 02 14:45:03.842701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.842632 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" event={"ID":"d0947e44-e37f-4591-adfe-714233cdbd98","Type":"ContainerDied","Data":"9ce4a10c142c132d70d08eb6f02262772c2345d5495f44391da6df11101457b0"} Nov 02 14:45:03.842871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.842856 5591 scope.go:122] "RemoveContainer" containerID="9ce4a10c142c132d70d08eb6f02262772c2345d5495f44391da6df11101457b0" Nov 02 14:45:03.844319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.844293 5591 generic.go:356] "Generic (PLEG): container finished" podID="cd174ecd-1998-4f54-8588-9de900552feb" containerID="02e1d759e376302c49fe32257feb0df18c35a52cc4cbffa2147d796d40d7bb5d" exitCode=60 Nov 02 14:45:03.844525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.844504 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" event={"ID":"cd174ecd-1998-4f54-8588-9de900552feb","Type":"ContainerDied","Data":"02e1d759e376302c49fe32257feb0df18c35a52cc4cbffa2147d796d40d7bb5d"} Nov 02 14:45:03.844665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.844651 5591 scope.go:122] "RemoveContainer" containerID="02e1d759e376302c49fe32257feb0df18c35a52cc4cbffa2147d796d40d7bb5d" Nov 02 14:45:03.846704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.846683 5591 generic.go:356] "Generic (PLEG): container finished" podID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerID="3596f01a759a4e679a481bc04e3d394b15afdbdc58177540782fe8c80409e182" exitCode=60 Nov 02 14:45:03.846806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.846707 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" event={"ID":"e59d93eb-2fa4-4094-98df-b0f1d75e0c94","Type":"ContainerDied","Data":"3596f01a759a4e679a481bc04e3d394b15afdbdc58177540782fe8c80409e182"} Nov 02 14:45:03.846973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.846955 5591 scope.go:122] "RemoveContainer" containerID="3596f01a759a4e679a481bc04e3d394b15afdbdc58177540782fe8c80409e182" Nov 02 14:45:03.848609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.848588 5591 generic.go:356] "Generic (PLEG): container finished" podID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerID="87e5f68e4440c3ade2b39821678eb882389080bf9e8defcd55c3ee95cc69f3f4" exitCode=60 Nov 02 14:45:03.848689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.848615 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" event={"ID":"70f48a2c-7440-4363-bfc0-a7ccafdbed80","Type":"ContainerDied","Data":"87e5f68e4440c3ade2b39821678eb882389080bf9e8defcd55c3ee95cc69f3f4"} Nov 02 14:45:03.848881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.848865 5591 scope.go:122] "RemoveContainer" containerID="87e5f68e4440c3ade2b39821678eb882389080bf9e8defcd55c3ee95cc69f3f4" Nov 02 14:45:03.850224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.850203 5591 generic.go:356] "Generic (PLEG): container finished" podID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerID="6576f3a061960edea2951a380ffb7efb1a655de728d2dd1e0b9513fc6edb073d" exitCode=60 Nov 02 14:45:03.850318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.850283 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" event={"ID":"df7a7fb7-de79-44d6-ac70-753b949d5a12","Type":"ContainerDied","Data":"6576f3a061960edea2951a380ffb7efb1a655de728d2dd1e0b9513fc6edb073d"} Nov 02 14:45:03.850466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.850451 5591 scope.go:122] "RemoveContainer" containerID="6576f3a061960edea2951a380ffb7efb1a655de728d2dd1e0b9513fc6edb073d" Nov 02 14:45:03.852029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.852009 5591 generic.go:356] "Generic (PLEG): container finished" podID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerID="7ee5a1fa0005d3db5365b6b5448bd53690e26e0dfadf6270bc948436a5ab4f22" exitCode=60 Nov 02 14:45:03.852114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.852077 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" event={"ID":"85f73d79-17ec-4ba0-a85c-08ba05fd9af6","Type":"ContainerDied","Data":"7ee5a1fa0005d3db5365b6b5448bd53690e26e0dfadf6270bc948436a5ab4f22"} Nov 02 14:45:03.852248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.852234 5591 scope.go:122] "RemoveContainer" containerID="7ee5a1fa0005d3db5365b6b5448bd53690e26e0dfadf6270bc948436a5ab4f22" Nov 02 14:45:03.853716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.853697 5591 generic.go:356] "Generic (PLEG): container finished" podID="96800fad-b801-4f01-91db-db678d597bf1" containerID="16a3f7b59015cddd69da0bb07f6bee054c1e541f38de14f50a11719c8039cd0b" exitCode=60 Nov 02 14:45:03.853806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.853716 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" event={"ID":"96800fad-b801-4f01-91db-db678d597bf1","Type":"ContainerDied","Data":"16a3f7b59015cddd69da0bb07f6bee054c1e541f38de14f50a11719c8039cd0b"} Nov 02 14:45:03.854025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.853973 5591 scope.go:122] "RemoveContainer" containerID="16a3f7b59015cddd69da0bb07f6bee054c1e541f38de14f50a11719c8039cd0b" Nov 02 14:45:03.855506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.855486 5591 generic.go:356] "Generic (PLEG): container finished" podID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerID="3b70c657ae37f9392a2893f6329fa6b9c6e733a8224c1b7bf8b142df3de9f0ac" exitCode=60 Nov 02 14:45:03.855610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.855565 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" event={"ID":"62221ac5-75fe-4999-9df5-2952cd3dee73","Type":"ContainerDied","Data":"3b70c657ae37f9392a2893f6329fa6b9c6e733a8224c1b7bf8b142df3de9f0ac"} Nov 02 14:45:03.855740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.855725 5591 scope.go:122] "RemoveContainer" containerID="3b70c657ae37f9392a2893f6329fa6b9c6e733a8224c1b7bf8b142df3de9f0ac" Nov 02 14:45:03.859108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.859087 5591 generic.go:356] "Generic (PLEG): container finished" podID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerID="15707fd1106a80f430d33f5a94ac9dddfb546c395d0f33c8e056b049b4d20133" exitCode=60 Nov 02 14:45:03.859213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.859167 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" event={"ID":"91775710-4a1d-4dbb-ab4d-d3c771f4cad2","Type":"ContainerDied","Data":"15707fd1106a80f430d33f5a94ac9dddfb546c395d0f33c8e056b049b4d20133"} Nov 02 14:45:03.859476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.859412 5591 scope.go:122] "RemoveContainer" containerID="15707fd1106a80f430d33f5a94ac9dddfb546c395d0f33c8e056b049b4d20133" Nov 02 14:45:03.860908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.860886 5591 generic.go:356] "Generic (PLEG): container finished" podID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerID="78f43a89fc7f2958f5fa603708870df6dcec8b8d84716606fc3739d307e8d4be" exitCode=60 Nov 02 14:45:03.861064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.860964 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" event={"ID":"b9ce129d-05f8-48db-82cb-fa836c2992a0","Type":"ContainerDied","Data":"78f43a89fc7f2958f5fa603708870df6dcec8b8d84716606fc3739d307e8d4be"} Nov 02 14:45:03.861224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.861209 5591 scope.go:122] "RemoveContainer" containerID="78f43a89fc7f2958f5fa603708870df6dcec8b8d84716606fc3739d307e8d4be" Nov 02 14:45:03.862713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.862692 5591 generic.go:356] "Generic (PLEG): container finished" podID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerID="30b52cbbef483b08ee11b76eb5e36268a8d1e681a057f8a3c5d315959dbb42c5" exitCode=60 Nov 02 14:45:03.862826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.862770 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" event={"ID":"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c","Type":"ContainerDied","Data":"30b52cbbef483b08ee11b76eb5e36268a8d1e681a057f8a3c5d315959dbb42c5"} Nov 02 14:45:03.863082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.863058 5591 scope.go:122] "RemoveContainer" containerID="30b52cbbef483b08ee11b76eb5e36268a8d1e681a057f8a3c5d315959dbb42c5" Nov 02 14:45:03.868063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.868035 5591 generic.go:356] "Generic (PLEG): container finished" podID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerID="894f16ad061f9f6d1d3f0a54c81fd15ed40c9f63a2485197fb9b36dfa5f4a36e" exitCode=60 Nov 02 14:45:03.868162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.868143 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" event={"ID":"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8","Type":"ContainerDied","Data":"894f16ad061f9f6d1d3f0a54c81fd15ed40c9f63a2485197fb9b36dfa5f4a36e"} Nov 02 14:45:03.868645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.868567 5591 scope.go:122] "RemoveContainer" containerID="894f16ad061f9f6d1d3f0a54c81fd15ed40c9f63a2485197fb9b36dfa5f4a36e" Nov 02 14:45:03.885591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:03.885545 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" podStartSLOduration=3.885530546 podStartE2EDuration="3.885530546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.882282529 +0000 UTC m=+3652.607904299" watchObservedRunningTime="2026-11-02 14:45:03.885530546 +0000 UTC m=+3652.611152316" Nov 02 14:45:04.224494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.224458 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:04.880562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.880527 5591 generic.go:356] "Generic (PLEG): container finished" podID="d6982912-1475-49a1-a605-2db8b5bc7687" containerID="fbc97338b4e8cef3445b8018071bc430421dc76e9c5a785efdefd03d9ae2a8f1" exitCode=60 Nov 02 14:45:04.881024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.880611 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" event={"ID":"d6982912-1475-49a1-a605-2db8b5bc7687","Type":"ContainerDied","Data":"fbc97338b4e8cef3445b8018071bc430421dc76e9c5a785efdefd03d9ae2a8f1"} Nov 02 14:45:04.881024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.880672 5591 scope.go:122] "RemoveContainer" containerID="3aa06552bb6980e808c3aa571ee4bcb6f7866d615f95a4edb729e6669f6ac1aa" Nov 02 14:45:04.881024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.880971 5591 scope.go:122] "RemoveContainer" containerID="fbc97338b4e8cef3445b8018071bc430421dc76e9c5a785efdefd03d9ae2a8f1" Nov 02 14:45:04.881403 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:04.881382 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5_odh-ai-gateway-infra(d6982912-1475-49a1-a605-2db8b5bc7687)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" Nov 02 14:45:04.882650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.882621 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" event={"ID":"d0947e44-e37f-4591-adfe-714233cdbd98","Type":"ContainerStarted","Data":"af10f9ae3635ec6e923427706d687f4c697e2ea83381a6ba268cced461cfcfac"} Nov 02 14:45:04.885089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.885064 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" event={"ID":"cd174ecd-1998-4f54-8588-9de900552feb","Type":"ContainerStarted","Data":"d65ae15a04f39bd29ce74aa3ace357ffd42041283060cb898aece01517af1c72"} Nov 02 14:45:04.888922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.888893 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" event={"ID":"e59d93eb-2fa4-4094-98df-b0f1d75e0c94","Type":"ContainerStarted","Data":"90aea3f8bbbf3aa3701d164df72d176a1415c4519006634ebc6c7a82f0f2ad45"} Nov 02 14:45:04.891435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.891404 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" event={"ID":"70f48a2c-7440-4363-bfc0-a7ccafdbed80","Type":"ContainerStarted","Data":"fb4c9f735e675eb74fa032d5d705b7bf2b0b2e07f429636c707267036ab26a54"} Nov 02 14:45:04.893541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.893518 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" event={"ID":"df7a7fb7-de79-44d6-ac70-753b949d5a12","Type":"ContainerStarted","Data":"6b48ef0bfa3965120311816b4de41671f596eb4abf707b0147524f2c3322dc0e"} Nov 02 14:45:04.895629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.895605 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" event={"ID":"85f73d79-17ec-4ba0-a85c-08ba05fd9af6","Type":"ContainerStarted","Data":"60c930aba41797368981ae827dc60cb33e018297c0145a8e43d13543c4aa3131"} Nov 02 14:45:04.897823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.897802 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" event={"ID":"96800fad-b801-4f01-91db-db678d597bf1","Type":"ContainerStarted","Data":"ce8db83fde1be3799b2163eb3db7fd6d544c2442c68b1190f7150f92ed4a56ca"} Nov 02 14:45:04.899865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.899841 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" event={"ID":"b9ce129d-05f8-48db-82cb-fa836c2992a0","Type":"ContainerStarted","Data":"595ecef24865126c8f5a2c53608abf8da20d8414d54c378c9b86f66649e912b4"} Nov 02 14:45:04.901813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.901792 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" event={"ID":"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c","Type":"ContainerStarted","Data":"265447ec69830b4ac811b022c2ff611ea821075aa5840572df369d539cb98670"} Nov 02 14:45:04.904013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:04.903973 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" event={"ID":"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8","Type":"ContainerStarted","Data":"b04b173a712f36421fd2b8e96ddc257b9169df48fd351144f491ad28cd5a7774"} Nov 02 14:45:05.086059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.086022 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:45:05.919669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.919630 5591 generic.go:356] "Generic (PLEG): container finished" podID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerID="3e6d09b805d5a08ebe53b368db4a2dc6acf6f414c1f82e899a2e3eac26186efe" exitCode=60 Nov 02 14:45:05.920222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.919716 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" event={"ID":"62221ac5-75fe-4999-9df5-2952cd3dee73","Type":"ContainerDied","Data":"3e6d09b805d5a08ebe53b368db4a2dc6acf6f414c1f82e899a2e3eac26186efe"} Nov 02 14:45:05.920222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.919781 5591 scope.go:122] "RemoveContainer" containerID="3b70c657ae37f9392a2893f6329fa6b9c6e733a8224c1b7bf8b142df3de9f0ac" Nov 02 14:45:05.920222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.920163 5591 scope.go:122] "RemoveContainer" containerID="3e6d09b805d5a08ebe53b368db4a2dc6acf6f414c1f82e899a2e3eac26186efe" Nov 02 14:45:05.920591 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.920564 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg_odh-ai-gateway-infra(62221ac5-75fe-4999-9df5-2952cd3dee73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" Nov 02 14:45:05.922178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.922151 5591 generic.go:356] "Generic (PLEG): container finished" podID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerID="33556771fc50391db0ca2463acb4741e9438f9f5f167beeeb0506427d7e8a597" exitCode=60 Nov 02 14:45:05.922278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.922226 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" event={"ID":"91775710-4a1d-4dbb-ab4d-d3c771f4cad2","Type":"ContainerDied","Data":"33556771fc50391db0ca2463acb4741e9438f9f5f167beeeb0506427d7e8a597"} Nov 02 14:45:05.922540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.922522 5591 scope.go:122] "RemoveContainer" containerID="33556771fc50391db0ca2463acb4741e9438f9f5f167beeeb0506427d7e8a597" Nov 02 14:45:05.922905 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.922885 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc_odh-ai-gateway-infra(91775710-4a1d-4dbb-ab4d-d3c771f4cad2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" Nov 02 14:45:05.924225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.924201 5591 generic.go:356] "Generic (PLEG): container finished" podID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerID="595ecef24865126c8f5a2c53608abf8da20d8414d54c378c9b86f66649e912b4" exitCode=60 Nov 02 14:45:05.924315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.924227 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" event={"ID":"b9ce129d-05f8-48db-82cb-fa836c2992a0","Type":"ContainerDied","Data":"595ecef24865126c8f5a2c53608abf8da20d8414d54c378c9b86f66649e912b4"} Nov 02 14:45:05.924579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.924558 5591 scope.go:122] "RemoveContainer" containerID="595ecef24865126c8f5a2c53608abf8da20d8414d54c378c9b86f66649e912b4" Nov 02 14:45:05.924929 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.924907 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l_odh-ai-gateway-infra(b9ce129d-05f8-48db-82cb-fa836c2992a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" Nov 02 14:45:05.925921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.925902 5591 generic.go:356] "Generic (PLEG): container finished" podID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerID="265447ec69830b4ac811b022c2ff611ea821075aa5840572df369d539cb98670" exitCode=60 Nov 02 14:45:05.926050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.925954 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" event={"ID":"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c","Type":"ContainerDied","Data":"265447ec69830b4ac811b022c2ff611ea821075aa5840572df369d539cb98670"} Nov 02 14:45:05.926287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.926272 5591 scope.go:122] "RemoveContainer" containerID="265447ec69830b4ac811b022c2ff611ea821075aa5840572df369d539cb98670" Nov 02 14:45:05.926499 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.926483 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw_odh-ai-gateway-infra(b7d03007-f6a2-47d4-8742-4ca9d4d2f71c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" Nov 02 14:45:05.928346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.928327 5591 generic.go:356] "Generic (PLEG): container finished" podID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerID="b04b173a712f36421fd2b8e96ddc257b9169df48fd351144f491ad28cd5a7774" exitCode=60 Nov 02 14:45:05.928433 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.928415 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" event={"ID":"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8","Type":"ContainerDied","Data":"b04b173a712f36421fd2b8e96ddc257b9169df48fd351144f491ad28cd5a7774"} Nov 02 14:45:05.928629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.928610 5591 scope.go:122] "RemoveContainer" containerID="b04b173a712f36421fd2b8e96ddc257b9169df48fd351144f491ad28cd5a7774" Nov 02 14:45:05.928921 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.928900 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn_odh-ai-gateway-infra(0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" Nov 02 14:45:05.930556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.930535 5591 scope.go:122] "RemoveContainer" containerID="fbc97338b4e8cef3445b8018071bc430421dc76e9c5a785efdefd03d9ae2a8f1" Nov 02 14:45:05.930777 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.930753 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5_odh-ai-gateway-infra(d6982912-1475-49a1-a605-2db8b5bc7687)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" Nov 02 14:45:05.932500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.932481 5591 generic.go:356] "Generic (PLEG): container finished" podID="d0947e44-e37f-4591-adfe-714233cdbd98" containerID="af10f9ae3635ec6e923427706d687f4c697e2ea83381a6ba268cced461cfcfac" exitCode=60 Nov 02 14:45:05.932607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.932554 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" event={"ID":"d0947e44-e37f-4591-adfe-714233cdbd98","Type":"ContainerDied","Data":"af10f9ae3635ec6e923427706d687f4c697e2ea83381a6ba268cced461cfcfac"} Nov 02 14:45:05.932781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.932762 5591 scope.go:122] "RemoveContainer" containerID="af10f9ae3635ec6e923427706d687f4c697e2ea83381a6ba268cced461cfcfac" Nov 02 14:45:05.933080 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.933056 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6_odh-ai-gateway-infra(d0947e44-e37f-4591-adfe-714233cdbd98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" Nov 02 14:45:05.938840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.937446 5591 scope.go:122] "RemoveContainer" containerID="15707fd1106a80f430d33f5a94ac9dddfb546c395d0f33c8e056b049b4d20133" Nov 02 14:45:05.938840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.938044 5591 generic.go:356] "Generic (PLEG): container finished" podID="cd174ecd-1998-4f54-8588-9de900552feb" containerID="d65ae15a04f39bd29ce74aa3ace357ffd42041283060cb898aece01517af1c72" exitCode=60 Nov 02 14:45:05.938840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.938153 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" event={"ID":"cd174ecd-1998-4f54-8588-9de900552feb","Type":"ContainerDied","Data":"d65ae15a04f39bd29ce74aa3ace357ffd42041283060cb898aece01517af1c72"} Nov 02 14:45:05.938840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.938490 5591 scope.go:122] "RemoveContainer" containerID="d65ae15a04f39bd29ce74aa3ace357ffd42041283060cb898aece01517af1c72" Nov 02 14:45:05.938840 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.938753 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42_odh-ai-gateway-infra(cd174ecd-1998-4f54-8588-9de900552feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" podUID="cd174ecd-1998-4f54-8588-9de900552feb" Nov 02 14:45:05.941473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.941450 5591 generic.go:356] "Generic (PLEG): container finished" podID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerID="90aea3f8bbbf3aa3701d164df72d176a1415c4519006634ebc6c7a82f0f2ad45" exitCode=60 Nov 02 14:45:05.941548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.941518 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" event={"ID":"e59d93eb-2fa4-4094-98df-b0f1d75e0c94","Type":"ContainerDied","Data":"90aea3f8bbbf3aa3701d164df72d176a1415c4519006634ebc6c7a82f0f2ad45"} Nov 02 14:45:05.941770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.941752 5591 scope.go:122] "RemoveContainer" containerID="90aea3f8bbbf3aa3701d164df72d176a1415c4519006634ebc6c7a82f0f2ad45" Nov 02 14:45:05.942087 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.942064 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p_odh-ai-gateway-infra(e59d93eb-2fa4-4094-98df-b0f1d75e0c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" Nov 02 14:45:05.943383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.943362 5591 generic.go:356] "Generic (PLEG): container finished" podID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerID="fb4c9f735e675eb74fa032d5d705b7bf2b0b2e07f429636c707267036ab26a54" exitCode=60 Nov 02 14:45:05.943502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.943445 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" event={"ID":"70f48a2c-7440-4363-bfc0-a7ccafdbed80","Type":"ContainerDied","Data":"fb4c9f735e675eb74fa032d5d705b7bf2b0b2e07f429636c707267036ab26a54"} Nov 02 14:45:05.943666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.943651 5591 scope.go:122] "RemoveContainer" containerID="fb4c9f735e675eb74fa032d5d705b7bf2b0b2e07f429636c707267036ab26a54" Nov 02 14:45:05.943896 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.943874 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl_odh-ai-gateway-infra(70f48a2c-7440-4363-bfc0-a7ccafdbed80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" Nov 02 14:45:05.945721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.945690 5591 generic.go:356] "Generic (PLEG): container finished" podID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerID="6b48ef0bfa3965120311816b4de41671f596eb4abf707b0147524f2c3322dc0e" exitCode=60 Nov 02 14:45:05.945813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.945773 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" event={"ID":"df7a7fb7-de79-44d6-ac70-753b949d5a12","Type":"ContainerDied","Data":"6b48ef0bfa3965120311816b4de41671f596eb4abf707b0147524f2c3322dc0e"} Nov 02 14:45:05.946010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.945994 5591 scope.go:122] "RemoveContainer" containerID="6b48ef0bfa3965120311816b4de41671f596eb4abf707b0147524f2c3322dc0e" Nov 02 14:45:05.946260 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.946241 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv_odh-ai-gateway-infra(df7a7fb7-de79-44d6-ac70-753b949d5a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" Nov 02 14:45:05.947948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.947927 5591 generic.go:356] "Generic (PLEG): container finished" podID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerID="60c930aba41797368981ae827dc60cb33e018297c0145a8e43d13543c4aa3131" exitCode=60 Nov 02 14:45:05.948059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.948017 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" event={"ID":"85f73d79-17ec-4ba0-a85c-08ba05fd9af6","Type":"ContainerDied","Data":"60c930aba41797368981ae827dc60cb33e018297c0145a8e43d13543c4aa3131"} Nov 02 14:45:05.948622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.948600 5591 scope.go:122] "RemoveContainer" containerID="60c930aba41797368981ae827dc60cb33e018297c0145a8e43d13543c4aa3131" Nov 02 14:45:05.948869 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.948835 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6_odh-ai-gateway-infra(85f73d79-17ec-4ba0-a85c-08ba05fd9af6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" Nov 02 14:45:05.949082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.949061 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:05.950258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.950235 5591 generic.go:356] "Generic (PLEG): container finished" podID="96800fad-b801-4f01-91db-db678d597bf1" containerID="ce8db83fde1be3799b2163eb3db7fd6d544c2442c68b1190f7150f92ed4a56ca" exitCode=60 Nov 02 14:45:05.950356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.950294 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" event={"ID":"96800fad-b801-4f01-91db-db678d597bf1","Type":"ContainerDied","Data":"ce8db83fde1be3799b2163eb3db7fd6d544c2442c68b1190f7150f92ed4a56ca"} Nov 02 14:45:05.950553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.950539 5591 scope.go:122] "RemoveContainer" containerID="ce8db83fde1be3799b2163eb3db7fd6d544c2442c68b1190f7150f92ed4a56ca" Nov 02 14:45:05.950820 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:05.950800 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv_odh-ai-gateway-infra(96800fad-b801-4f01-91db-db678d597bf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" podUID="96800fad-b801-4f01-91db-db678d597bf1" Nov 02 14:45:05.955892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.955874 5591 scope.go:122] "RemoveContainer" containerID="78f43a89fc7f2958f5fa603708870df6dcec8b8d84716606fc3739d307e8d4be" Nov 02 14:45:05.970360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.970336 5591 scope.go:122] "RemoveContainer" containerID="30b52cbbef483b08ee11b76eb5e36268a8d1e681a057f8a3c5d315959dbb42c5" Nov 02 14:45:05.984604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.984582 5591 scope.go:122] "RemoveContainer" containerID="894f16ad061f9f6d1d3f0a54c81fd15ed40c9f63a2485197fb9b36dfa5f4a36e" Nov 02 14:45:05.999125 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:05.999102 5591 scope.go:122] "RemoveContainer" containerID="9ce4a10c142c132d70d08eb6f02262772c2345d5495f44391da6df11101457b0" Nov 02 14:45:06.012889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:06.012871 5591 scope.go:122] "RemoveContainer" containerID="02e1d759e376302c49fe32257feb0df18c35a52cc4cbffa2147d796d40d7bb5d" Nov 02 14:45:06.029310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:06.029285 5591 scope.go:122] "RemoveContainer" containerID="3596f01a759a4e679a481bc04e3d394b15afdbdc58177540782fe8c80409e182" Nov 02 14:45:06.044778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:06.044752 5591 scope.go:122] "RemoveContainer" containerID="87e5f68e4440c3ade2b39821678eb882389080bf9e8defcd55c3ee95cc69f3f4" Nov 02 14:45:06.059526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:06.059502 5591 scope.go:122] "RemoveContainer" containerID="6576f3a061960edea2951a380ffb7efb1a655de728d2dd1e0b9513fc6edb073d" Nov 02 14:45:06.074047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:06.074024 5591 scope.go:122] "RemoveContainer" containerID="7ee5a1fa0005d3db5365b6b5448bd53690e26e0dfadf6270bc948436a5ab4f22" Nov 02 14:45:06.089397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:06.089372 5591 scope.go:122] "RemoveContainer" containerID="16a3f7b59015cddd69da0bb07f6bee054c1e541f38de14f50a11719c8039cd0b" Nov 02 14:45:07.359863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.359830 5591 scope.go:122] "RemoveContainer" containerID="3e6d09b805d5a08ebe53b368db4a2dc6acf6f414c1f82e899a2e3eac26186efe" Nov 02 14:45:07.362046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.362017 5591 scope.go:122] "RemoveContainer" containerID="33556771fc50391db0ca2463acb4741e9438f9f5f167beeeb0506427d7e8a597" Nov 02 14:45:07.362455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.362430 5591 scope.go:122] "RemoveContainer" containerID="595ecef24865126c8f5a2c53608abf8da20d8414d54c378c9b86f66649e912b4" Nov 02 14:45:07.362888 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.362854 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc_odh-ai-gateway-infra(91775710-4a1d-4dbb-ab4d-d3c771f4cad2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" Nov 02 14:45:07.363024 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.362511 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg_odh-ai-gateway-infra(62221ac5-75fe-4999-9df5-2952cd3dee73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" Nov 02 14:45:07.363552 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.363526 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l_odh-ai-gateway-infra(b9ce129d-05f8-48db-82cb-fa836c2992a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" Nov 02 14:45:07.370295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.370273 5591 scope.go:122] "RemoveContainer" containerID="265447ec69830b4ac811b022c2ff611ea821075aa5840572df369d539cb98670" Nov 02 14:45:07.370910 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.370886 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw_odh-ai-gateway-infra(b7d03007-f6a2-47d4-8742-4ca9d4d2f71c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" Nov 02 14:45:07.373935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.373902 5591 scope.go:122] "RemoveContainer" containerID="b04b173a712f36421fd2b8e96ddc257b9169df48fd351144f491ad28cd5a7774" Nov 02 14:45:07.374611 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.374582 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn_odh-ai-gateway-infra(0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" Nov 02 14:45:07.377161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.377138 5591 scope.go:122] "RemoveContainer" containerID="af10f9ae3635ec6e923427706d687f4c697e2ea83381a6ba268cced461cfcfac" Nov 02 14:45:07.378031 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.377745 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6_odh-ai-gateway-infra(d0947e44-e37f-4591-adfe-714233cdbd98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" Nov 02 14:45:07.379485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.379465 5591 scope.go:122] "RemoveContainer" containerID="d65ae15a04f39bd29ce74aa3ace357ffd42041283060cb898aece01517af1c72" Nov 02 14:45:07.380206 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.380178 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42_odh-ai-gateway-infra(cd174ecd-1998-4f54-8588-9de900552feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" podUID="cd174ecd-1998-4f54-8588-9de900552feb" Nov 02 14:45:07.383217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.383198 5591 scope.go:122] "RemoveContainer" containerID="90aea3f8bbbf3aa3701d164df72d176a1415c4519006634ebc6c7a82f0f2ad45" Nov 02 14:45:07.404871 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.404828 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p_odh-ai-gateway-infra(e59d93eb-2fa4-4094-98df-b0f1d75e0c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" Nov 02 14:45:07.418568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.418386 5591 scope.go:122] "RemoveContainer" containerID="fb4c9f735e675eb74fa032d5d705b7bf2b0b2e07f429636c707267036ab26a54" Nov 02 14:45:07.419141 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.419108 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl_odh-ai-gateway-infra(70f48a2c-7440-4363-bfc0-a7ccafdbed80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" Nov 02 14:45:07.422404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.422376 5591 scope.go:122] "RemoveContainer" containerID="6b48ef0bfa3965120311816b4de41671f596eb4abf707b0147524f2c3322dc0e" Nov 02 14:45:07.424465 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.423094 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv_odh-ai-gateway-infra(df7a7fb7-de79-44d6-ac70-753b949d5a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" Nov 02 14:45:07.426910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.426890 5591 scope.go:122] "RemoveContainer" containerID="60c930aba41797368981ae827dc60cb33e018297c0145a8e43d13543c4aa3131" Nov 02 14:45:07.430467 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.430437 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6_odh-ai-gateway-infra(85f73d79-17ec-4ba0-a85c-08ba05fd9af6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" Nov 02 14:45:07.430628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.430605 5591 scope.go:122] "RemoveContainer" containerID="ce8db83fde1be3799b2163eb3db7fd6d544c2442c68b1190f7150f92ed4a56ca" Nov 02 14:45:07.431032 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:07.430976 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv_odh-ai-gateway-infra(96800fad-b801-4f01-91db-db678d597bf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" podUID="96800fad-b801-4f01-91db-db678d597bf1" Nov 02 14:45:07.698147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:07.698102 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:09.475796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:09.475754 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:11.263009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:11.259216 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:11.544342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:11.543239 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn"] Nov 02 14:45:11.544342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:11.543466 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" containerID="cri-o://a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1" gracePeriod=30 Nov 02 14:45:12.950008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:12.949143 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:13.200267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:13.200177 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" podUID="ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:45:14.091219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.091190 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:45:14.118840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.118804 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-plugins-config-volume\") pod \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\" (UID: \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\") " Nov 02 14:45:14.119060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.119046 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-kube-api-access-xvw7n\" (UniqueName: \"kubernetes.io/projected/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-kube-api-access-xvw7n\") pod \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\" (UID: \"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c\") " Nov 02 14:45:14.119247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.119222 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-plugins-config-volume" pod "ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" (UID: "ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:14.119623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.119601 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:14.121183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.121160 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-kube-api-access-xvw7n" pod "ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" (UID: "ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c"). InnerVolumeSpecName "kube-api-access-xvw7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:14.194810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.194769 5591 scope.go:122] "RemoveContainer" containerID="0235655b7fa965f77c7fb07989a608883a5770a333ca4f2a0e4c5cf29860d27d" Nov 02 14:45:14.221045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.221013 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvw7n\" (UniqueName: \"kubernetes.io/projected/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c-kube-api-access-xvw7n\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:14.413624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.413588 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:14.629005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.626093 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" event={"ID":"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c","Type":"ContainerDied","Data":"0235655b7fa965f77c7fb07989a608883a5770a333ca4f2a0e4c5cf29860d27d"} Nov 02 14:45:14.629005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.626560 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" Nov 02 14:45:14.629005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.629069 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n" event={"ID":"ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c","Type":"ContainerDied","Data":"9fc82279277dcd0ce5b7bc4c2ae8ecd3941a5bb74d82684f3bbe1373a8fb57e9"} Nov 02 14:45:14.871007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.862864 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:14.916078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.911347 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n"] Nov 02 14:45:14.930903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:14.927349 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1fcb4-5d65895fc5-2zs9n"] Nov 02 14:45:15.432019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:15.429797 5591 scope.go:122] "RemoveContainer" containerID="fbc97338b4e8cef3445b8018071bc430421dc76e9c5a785efdefd03d9ae2a8f1" Nov 02 14:45:15.450067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:15.450039 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" path="/var/lib/kubelet/pods/ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c/volumes" Nov 02 14:45:15.619521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:15.619484 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:16.384361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:16.384319 5591 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:45:16.384620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:16.384380 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:45:16.423092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:16.423006 5591 scope.go:122] "RemoveContainer" containerID="ce8db83fde1be3799b2163eb3db7fd6d544c2442c68b1190f7150f92ed4a56ca" Nov 02 14:45:16.423280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:16.423170 5591 scope.go:122] "RemoveContainer" containerID="b04b173a712f36421fd2b8e96ddc257b9169df48fd351144f491ad28cd5a7774" Nov 02 14:45:16.423348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:16.423331 5591 scope.go:122] "RemoveContainer" containerID="6b48ef0bfa3965120311816b4de41671f596eb4abf707b0147524f2c3322dc0e" Nov 02 14:45:16.659594 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:16.659556 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" event={"ID":"d6982912-1475-49a1-a605-2db8b5bc7687","Type":"ContainerStarted","Data":"cae206947c3823aeb748af3858e72cd550aa972b1d5f95061d38075b7f3386c5"} Nov 02 14:45:16.670145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:16.670117 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:16.847533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:16.847504 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:17.029549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.029519 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:45:17.111804 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.111770 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4529748-2a47-4b75-b885-0a83f37a0705-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4529748-2a47-4b75-b885-0a83f37a0705-istiod-ca-cert\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.112001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.111943 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-certs\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.112058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112000 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-credential-socket\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.112058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112038 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-envoy\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.112152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112109 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4529748-2a47-4b75-b885-0a83f37a0705-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4529748-2a47-4b75-b885-0a83f37a0705-istio-podinfo\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.112286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112266 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-istio-token\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-istio-token\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.112382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112277 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-certs" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:17.112382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112355 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-socket\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.112500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112385 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-credential-socket" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:17.112500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112393 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4529748-2a47-4b75-b885-0a83f37a0705-istiod-ca-cert" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:17.112650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112626 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-socket" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:17.112731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112685 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-kube-api-access-zgkm4\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-kube-api-access-zgkm4\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.112794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.112749 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-data\") pod \"b4529748-2a47-4b75-b885-0a83f37a0705\" (UID: \"b4529748-2a47-4b75-b885-0a83f37a0705\") " Nov 02 14:45:17.113335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.113307 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-data" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:17.113640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.113618 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.113751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.113645 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.113751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.113659 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4529748-2a47-4b75-b885-0a83f37a0705-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.113751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.113674 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.113751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.113689 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.114786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.114754 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4529748-2a47-4b75-b885-0a83f37a0705-istio-podinfo" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:45:17.114888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.114838 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-envoy" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:17.115050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.115028 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-istio-token" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:17.115550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.115532 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-kube-api-access-zgkm4" pod "b4529748-2a47-4b75-b885-0a83f37a0705" (UID: "b4529748-2a47-4b75-b885-0a83f37a0705"). InnerVolumeSpecName "kube-api-access-zgkm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:17.215651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.215570 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgkm4\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-kube-api-access-zgkm4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.215651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.215601 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4529748-2a47-4b75-b885-0a83f37a0705-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.215651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.215613 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4529748-2a47-4b75-b885-0a83f37a0705-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.215651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.215623 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4529748-2a47-4b75-b885-0a83f37a0705-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:17.308007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.307651 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:17.436686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.436658 5591 scope.go:122] "RemoveContainer" containerID="60c930aba41797368981ae827dc60cb33e018297c0145a8e43d13543c4aa3131" Nov 02 14:45:17.436950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.436820 5591 scope.go:122] "RemoveContainer" containerID="265447ec69830b4ac811b022c2ff611ea821075aa5840572df369d539cb98670" Nov 02 14:45:17.516390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.511805 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:17.671447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.671410 5591 generic.go:356] "Generic (PLEG): container finished" podID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerID="4ea18935f1a48e4b0a603023d75cb50cfc49677041e3aa44a281edb217d7de57" exitCode=60 Nov 02 14:45:17.671870 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.671478 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" event={"ID":"df7a7fb7-de79-44d6-ac70-753b949d5a12","Type":"ContainerDied","Data":"4ea18935f1a48e4b0a603023d75cb50cfc49677041e3aa44a281edb217d7de57"} Nov 02 14:45:17.671870 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.671536 5591 scope.go:122] "RemoveContainer" containerID="6b48ef0bfa3965120311816b4de41671f596eb4abf707b0147524f2c3322dc0e" Nov 02 14:45:17.672050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.672033 5591 scope.go:122] "RemoveContainer" containerID="4ea18935f1a48e4b0a603023d75cb50cfc49677041e3aa44a281edb217d7de57" Nov 02 14:45:17.672489 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:17.672457 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv_odh-ai-gateway-infra(df7a7fb7-de79-44d6-ac70-753b949d5a12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" Nov 02 14:45:17.673818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.673797 5591 generic.go:356] "Generic (PLEG): container finished" podID="96800fad-b801-4f01-91db-db678d597bf1" containerID="98acd6787fabe97fd8dc7e256f922c105e98c43b6d2231a0446e07eee4571de7" exitCode=60 Nov 02 14:45:17.673936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.673866 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" event={"ID":"96800fad-b801-4f01-91db-db678d597bf1","Type":"ContainerDied","Data":"98acd6787fabe97fd8dc7e256f922c105e98c43b6d2231a0446e07eee4571de7"} Nov 02 14:45:17.674290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.674273 5591 scope.go:122] "RemoveContainer" containerID="98acd6787fabe97fd8dc7e256f922c105e98c43b6d2231a0446e07eee4571de7" Nov 02 14:45:17.674689 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:17.674667 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv_odh-ai-gateway-infra(96800fad-b801-4f01-91db-db678d597bf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" podUID="96800fad-b801-4f01-91db-db678d597bf1" Nov 02 14:45:17.675912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.675889 5591 generic.go:356] "Generic (PLEG): container finished" podID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerID="99eb9d159d055ef3f47ef4c7ffb1f1b9d6a1d4872131ffadcadcdb868adcdef4" exitCode=60 Nov 02 14:45:17.676029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.675946 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" event={"ID":"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8","Type":"ContainerDied","Data":"99eb9d159d055ef3f47ef4c7ffb1f1b9d6a1d4872131ffadcadcdb868adcdef4"} Nov 02 14:45:17.676392 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.676366 5591 scope.go:122] "RemoveContainer" containerID="99eb9d159d055ef3f47ef4c7ffb1f1b9d6a1d4872131ffadcadcdb868adcdef4" Nov 02 14:45:17.676749 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:17.676724 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn_odh-ai-gateway-infra(0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" Nov 02 14:45:17.677508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.677487 5591 generic.go:356] "Generic (PLEG): container finished" podID="b4529748-2a47-4b75-b885-0a83f37a0705" containerID="a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1" exitCode=0 Nov 02 14:45:17.677590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.677528 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" Nov 02 14:45:17.677590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.677566 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" event={"ID":"b4529748-2a47-4b75-b885-0a83f37a0705","Type":"ContainerDied","Data":"a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1"} Nov 02 14:45:17.677667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.677613 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn" event={"ID":"b4529748-2a47-4b75-b885-0a83f37a0705","Type":"ContainerDied","Data":"31131d67094d7d7007e262bd617536fa8523b91ab7531e607000cfb5b9bcf210"} Nov 02 14:45:17.679643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.679625 5591 generic.go:356] "Generic (PLEG): container finished" podID="d6982912-1475-49a1-a605-2db8b5bc7687" containerID="cae206947c3823aeb748af3858e72cd550aa972b1d5f95061d38075b7f3386c5" exitCode=60 Nov 02 14:45:17.679735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.679658 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" event={"ID":"d6982912-1475-49a1-a605-2db8b5bc7687","Type":"ContainerDied","Data":"cae206947c3823aeb748af3858e72cd550aa972b1d5f95061d38075b7f3386c5"} Nov 02 14:45:17.679943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.679930 5591 scope.go:122] "RemoveContainer" containerID="cae206947c3823aeb748af3858e72cd550aa972b1d5f95061d38075b7f3386c5" Nov 02 14:45:17.680205 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:17.680185 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5_odh-ai-gateway-infra(d6982912-1475-49a1-a605-2db8b5bc7687)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" Nov 02 14:45:17.716904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.716871 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5"] Nov 02 14:45:17.769107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.769072 5591 scope.go:122] "RemoveContainer" containerID="ce8db83fde1be3799b2163eb3db7fd6d544c2442c68b1190f7150f92ed4a56ca" Nov 02 14:45:17.787259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.787236 5591 scope.go:122] "RemoveContainer" containerID="b04b173a712f36421fd2b8e96ddc257b9169df48fd351144f491ad28cd5a7774" Nov 02 14:45:17.809265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.809237 5591 scope.go:122] "RemoveContainer" containerID="a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1" Nov 02 14:45:17.820857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.820832 5591 scope.go:122] "RemoveContainer" containerID="a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1" Nov 02 14:45:17.821200 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:17.821176 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1\": container with ID starting with a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1 not found: ID does not exist" containerID="a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1" Nov 02 14:45:17.821259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.821209 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1"} err="failed to get container status \"a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1\": rpc error: code = NotFound desc = could not find container \"a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1\": container with ID starting with a36981559d3b342bb2a520abb96c4278dc6d7caa10e9e136dfa9f8bb503c69a1 not found: ID does not exist" Nov 02 14:45:17.821259 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.821228 5591 scope.go:122] "RemoveContainer" containerID="fbc97338b4e8cef3445b8018071bc430421dc76e9c5a785efdefd03d9ae2a8f1" Nov 02 14:45:17.920872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.918964 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn"] Nov 02 14:45:17.924244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.924213 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-69rmn"] Nov 02 14:45:17.933236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.933203 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2"] Nov 02 14:45:17.934707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:17.933467 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" containerID="cri-o://c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851" gracePeriod=30 Nov 02 14:45:18.027840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.027479 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:18.423129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.423097 5591 scope.go:122] "RemoveContainer" containerID="af10f9ae3635ec6e923427706d687f4c697e2ea83381a6ba268cced461cfcfac" Nov 02 14:45:18.423322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.423282 5591 scope.go:122] "RemoveContainer" containerID="90aea3f8bbbf3aa3701d164df72d176a1415c4519006634ebc6c7a82f0f2ad45" Nov 02 14:45:18.423399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.423374 5591 scope.go:122] "RemoveContainer" containerID="595ecef24865126c8f5a2c53608abf8da20d8414d54c378c9b86f66649e912b4" Nov 02 14:45:18.423652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.423624 5591 scope.go:122] "RemoveContainer" containerID="3e6d09b805d5a08ebe53b368db4a2dc6acf6f414c1f82e899a2e3eac26186efe" Nov 02 14:45:18.689967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.689899 5591 generic.go:356] "Generic (PLEG): container finished" podID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerID="65d7820b67d70fff2c9ea8d8d194b8396258ebc987fde08c35d955c952542a00" exitCode=60 Nov 02 14:45:18.690352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.689968 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" event={"ID":"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c","Type":"ContainerDied","Data":"65d7820b67d70fff2c9ea8d8d194b8396258ebc987fde08c35d955c952542a00"} Nov 02 14:45:18.690352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.690053 5591 scope.go:122] "RemoveContainer" containerID="265447ec69830b4ac811b022c2ff611ea821075aa5840572df369d539cb98670" Nov 02 14:45:18.690579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.690550 5591 scope.go:122] "RemoveContainer" containerID="65d7820b67d70fff2c9ea8d8d194b8396258ebc987fde08c35d955c952542a00" Nov 02 14:45:18.691012 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:18.690976 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw_odh-ai-gateway-infra(b7d03007-f6a2-47d4-8742-4ca9d4d2f71c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" Nov 02 14:45:18.698774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.698746 5591 generic.go:356] "Generic (PLEG): container finished" podID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerID="5967922dfefae38b19fc87519f36806336a37cecdb27445ceea6e7cb3d8d1b37" exitCode=60 Nov 02 14:45:18.698903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.698820 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" event={"ID":"85f73d79-17ec-4ba0-a85c-08ba05fd9af6","Type":"ContainerDied","Data":"5967922dfefae38b19fc87519f36806336a37cecdb27445ceea6e7cb3d8d1b37"} Nov 02 14:45:18.699318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.699295 5591 scope.go:122] "RemoveContainer" containerID="5967922dfefae38b19fc87519f36806336a37cecdb27445ceea6e7cb3d8d1b37" Nov 02 14:45:18.699729 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:18.699707 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6_odh-ai-gateway-infra(85f73d79-17ec-4ba0-a85c-08ba05fd9af6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" Nov 02 14:45:18.734854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.734814 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv"] Nov 02 14:45:18.757806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.757778 5591 scope.go:122] "RemoveContainer" containerID="60c930aba41797368981ae827dc60cb33e018297c0145a8e43d13543c4aa3131" Nov 02 14:45:18.797846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.794719 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv"] Nov 02 14:45:18.834718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.824755 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn"] Nov 02 14:45:18.955143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:18.952907 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:19.239955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.239925 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:19.247564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.247540 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:19.252861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.252838 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:19.258169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.258151 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:19.262095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.262077 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:19.360621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.360586 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-maas-api-service-ca\") pod \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\" (UID: \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\") " Nov 02 14:45:19.360821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.360688 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-kube-api-access-wcrdn\" (UniqueName: \"kubernetes.io/projected/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-kube-api-access-wcrdn\") pod \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\" (UID: \"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8\") " Nov 02 14:45:19.360821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.360789 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7a7fb7-de79-44d6-ac70-753b949d5a12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7a7fb7-de79-44d6-ac70-753b949d5a12-maas-api-service-ca\") pod \"df7a7fb7-de79-44d6-ac70-753b949d5a12\" (UID: \"df7a7fb7-de79-44d6-ac70-753b949d5a12\") " Nov 02 14:45:19.360955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.360865 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96800fad-b801-4f01-91db-db678d597bf1-kube-api-access-89dp6\" (UniqueName: \"kubernetes.io/projected/96800fad-b801-4f01-91db-db678d597bf1-kube-api-access-89dp6\") pod \"96800fad-b801-4f01-91db-db678d597bf1\" (UID: \"96800fad-b801-4f01-91db-db678d597bf1\") " Nov 02 14:45:19.360955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.360905 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6982912-1475-49a1-a605-2db8b5bc7687-kube-api-access-ksjcw\" (UniqueName: \"kubernetes.io/projected/d6982912-1475-49a1-a605-2db8b5bc7687-kube-api-access-ksjcw\") pod \"d6982912-1475-49a1-a605-2db8b5bc7687\" (UID: \"d6982912-1475-49a1-a605-2db8b5bc7687\") " Nov 02 14:45:19.361102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.361038 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-maas-api-service-ca" pod "0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" (UID: "0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:19.361102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.361060 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6982912-1475-49a1-a605-2db8b5bc7687-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6982912-1475-49a1-a605-2db8b5bc7687-maas-api-service-ca\") pod \"d6982912-1475-49a1-a605-2db8b5bc7687\" (UID: \"d6982912-1475-49a1-a605-2db8b5bc7687\") " Nov 02 14:45:19.361227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.361166 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7a7fb7-de79-44d6-ac70-753b949d5a12-kube-api-access-gbx7j\" (UniqueName: \"kubernetes.io/projected/df7a7fb7-de79-44d6-ac70-753b949d5a12-kube-api-access-gbx7j\") pod \"df7a7fb7-de79-44d6-ac70-753b949d5a12\" (UID: \"df7a7fb7-de79-44d6-ac70-753b949d5a12\") " Nov 02 14:45:19.361294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.361240 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96800fad-b801-4f01-91db-db678d597bf1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96800fad-b801-4f01-91db-db678d597bf1-maas-api-service-ca\") pod \"96800fad-b801-4f01-91db-db678d597bf1\" (UID: \"96800fad-b801-4f01-91db-db678d597bf1\") " Nov 02 14:45:19.362279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.361832 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6982912-1475-49a1-a605-2db8b5bc7687-maas-api-service-ca" pod "d6982912-1475-49a1-a605-2db8b5bc7687" (UID: "d6982912-1475-49a1-a605-2db8b5bc7687"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:19.362279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.362202 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96800fad-b801-4f01-91db-db678d597bf1-maas-api-service-ca" pod "96800fad-b801-4f01-91db-db678d597bf1" (UID: "96800fad-b801-4f01-91db-db678d597bf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:19.362279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.362230 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:19.362279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.362255 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6982912-1475-49a1-a605-2db8b5bc7687-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:19.362895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.362873 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7a7fb7-de79-44d6-ac70-753b949d5a12-maas-api-service-ca" pod "df7a7fb7-de79-44d6-ac70-753b949d5a12" (UID: "df7a7fb7-de79-44d6-ac70-753b949d5a12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:19.363676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.363651 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6982912-1475-49a1-a605-2db8b5bc7687-kube-api-access-ksjcw" pod "d6982912-1475-49a1-a605-2db8b5bc7687" (UID: "d6982912-1475-49a1-a605-2db8b5bc7687"). InnerVolumeSpecName "kube-api-access-ksjcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:19.363909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.363880 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7a7fb7-de79-44d6-ac70-753b949d5a12-kube-api-access-gbx7j" pod "df7a7fb7-de79-44d6-ac70-753b949d5a12" (UID: "df7a7fb7-de79-44d6-ac70-753b949d5a12"). InnerVolumeSpecName "kube-api-access-gbx7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:19.365197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.365173 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-kube-api-access-wcrdn" pod "0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" (UID: "0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8"). InnerVolumeSpecName "kube-api-access-wcrdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:19.365409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.365389 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96800fad-b801-4f01-91db-db678d597bf1-kube-api-access-89dp6" pod "96800fad-b801-4f01-91db-db678d597bf1" (UID: "96800fad-b801-4f01-91db-db678d597bf1"). InnerVolumeSpecName "kube-api-access-89dp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:19.443567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.443534 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" path="/var/lib/kubelet/pods/b4529748-2a47-4b75-b885-0a83f37a0705/volumes" Nov 02 14:45:19.445974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.445932 5591 scope.go:122] "RemoveContainer" containerID="d65ae15a04f39bd29ce74aa3ace357ffd42041283060cb898aece01517af1c72" Nov 02 14:45:19.446238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.446108 5591 scope.go:122] "RemoveContainer" containerID="33556771fc50391db0ca2463acb4741e9438f9f5f167beeeb0506427d7e8a597" Nov 02 14:45:19.447242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.447180 5591 scope.go:122] "RemoveContainer" containerID="fb4c9f735e675eb74fa032d5d705b7bf2b0b2e07f429636c707267036ab26a54" Nov 02 14:45:19.466790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.465270 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dp6\" (UniqueName: \"kubernetes.io/projected/96800fad-b801-4f01-91db-db678d597bf1-kube-api-access-89dp6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:19.466790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.465321 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksjcw\" (UniqueName: \"kubernetes.io/projected/d6982912-1475-49a1-a605-2db8b5bc7687-kube-api-access-ksjcw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:19.466790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.465338 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbx7j\" (UniqueName: \"kubernetes.io/projected/df7a7fb7-de79-44d6-ac70-753b949d5a12-kube-api-access-gbx7j\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:19.466790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.465354 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96800fad-b801-4f01-91db-db678d597bf1-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:19.466790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.465369 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrdn\" (UniqueName: \"kubernetes.io/projected/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8-kube-api-access-wcrdn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:19.466790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.465391 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7a7fb7-de79-44d6-ac70-753b949d5a12-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:19.711284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.711247 5591 generic.go:356] "Generic (PLEG): container finished" podID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerID="cd21cd52d09cdbadbd3a4919d6ab2e233b11f8848750e3b6375fb5bc66a73e1f" exitCode=60 Nov 02 14:45:19.711707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.711316 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" event={"ID":"62221ac5-75fe-4999-9df5-2952cd3dee73","Type":"ContainerDied","Data":"cd21cd52d09cdbadbd3a4919d6ab2e233b11f8848750e3b6375fb5bc66a73e1f"} Nov 02 14:45:19.711707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.711416 5591 scope.go:122] "RemoveContainer" containerID="3e6d09b805d5a08ebe53b368db4a2dc6acf6f414c1f82e899a2e3eac26186efe" Nov 02 14:45:19.711872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.711854 5591 scope.go:122] "RemoveContainer" containerID="cd21cd52d09cdbadbd3a4919d6ab2e233b11f8848750e3b6375fb5bc66a73e1f" Nov 02 14:45:19.712323 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:19.712295 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg_odh-ai-gateway-infra(62221ac5-75fe-4999-9df5-2952cd3dee73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" Nov 02 14:45:19.713492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.713470 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" event={"ID":"b9ce129d-05f8-48db-82cb-fa836c2992a0","Type":"ContainerStarted","Data":"f4ce287ca578ee75ae9a32e79ae13aa77b1d239979a699257982f8c46aac5f3e"} Nov 02 14:45:19.716572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.716548 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" Nov 02 14:45:19.716685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.716553 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn" event={"ID":"0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8","Type":"ContainerDied","Data":"f23f8c94078f4e9747c5f872fc6c93326cd20a6954c70cfe08b3b9fe9554647b"} Nov 02 14:45:19.717907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.717885 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" Nov 02 14:45:19.717907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.717895 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5" event={"ID":"d6982912-1475-49a1-a605-2db8b5bc7687","Type":"ContainerDied","Data":"8ad1695987ff744f3fba657bb77a1f1e637e4c9cb62080e0b92e10c47b6aafda"} Nov 02 14:45:19.719821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.719799 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" event={"ID":"d0947e44-e37f-4591-adfe-714233cdbd98","Type":"ContainerStarted","Data":"65f1da06b22141139022160c079d68504784b4e1392b7bc70cc8e55fce0a02eb"} Nov 02 14:45:19.721609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.721588 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" event={"ID":"e59d93eb-2fa4-4094-98df-b0f1d75e0c94","Type":"ContainerStarted","Data":"f08b4de40e181c569a2b2d92c2518560f708dacdb647c139921903b0eafd190b"} Nov 02 14:45:19.722885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.722869 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" Nov 02 14:45:19.723007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.722902 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv" event={"ID":"df7a7fb7-de79-44d6-ac70-753b949d5a12","Type":"ContainerDied","Data":"fa7db9d2d94fd90b3dfdb5db588b4eec23835d39542eaba2f12857857cf0efaa"} Nov 02 14:45:19.725612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.725593 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" event={"ID":"96800fad-b801-4f01-91db-db678d597bf1","Type":"ContainerDied","Data":"ac47c4837f3f02b4d4fa29fd689160f3609749cae17c41f54e2bff49cb455311"} Nov 02 14:45:19.725695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.725602 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv" Nov 02 14:45:19.753764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.753738 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw"] Nov 02 14:45:19.859124 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.859086 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6"] Nov 02 14:45:19.991417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:19.991377 5591 scope.go:122] "RemoveContainer" containerID="99eb9d159d055ef3f47ef4c7ffb1f1b9d6a1d4872131ffadcadcdb868adcdef4" Nov 02 14:45:20.006632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.006543 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:20.035931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.035900 5591 scope.go:122] "RemoveContainer" containerID="cae206947c3823aeb748af3858e72cd550aa972b1d5f95061d38075b7f3386c5" Nov 02 14:45:20.075513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.075482 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-kube-api-access-gs7gm\" (UniqueName: \"kubernetes.io/projected/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-kube-api-access-gs7gm\") pod \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\" (UID: \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\") " Nov 02 14:45:20.075868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.075845 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-maas-api-service-ca\") pod \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\" (UID: \"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c\") " Nov 02 14:45:20.076207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.076182 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-maas-api-service-ca" pod "b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" (UID: "b7d03007-f6a2-47d4-8742-4ca9d4d2f71c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:20.076559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.076541 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:20.089677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.089642 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5"] Nov 02 14:45:20.094081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.094052 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-662e3e-29893845-tkvj5"] Nov 02 14:45:20.135420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.135380 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-kube-api-access-gs7gm" pod "b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" (UID: "b7d03007-f6a2-47d4-8742-4ca9d4d2f71c"). InnerVolumeSpecName "kube-api-access-gs7gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:20.145010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.144973 5591 scope.go:122] "RemoveContainer" containerID="4ea18935f1a48e4b0a603023d75cb50cfc49677041e3aa44a281edb217d7de57" Nov 02 14:45:20.147997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.147952 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:20.156219 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.156171 5591 scope.go:122] "RemoveContainer" containerID="98acd6787fabe97fd8dc7e256f922c105e98c43b6d2231a0446e07eee4571de7" Nov 02 14:45:20.177236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.177201 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-maas-api-service-ca\") pod \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\" (UID: \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\") " Nov 02 14:45:20.177509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.177442 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-kube-api-access-kgjq7\" (UniqueName: \"kubernetes.io/projected/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-kube-api-access-kgjq7\") pod \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\" (UID: \"85f73d79-17ec-4ba0-a85c-08ba05fd9af6\") " Nov 02 14:45:20.177991 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.177945 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-maas-api-service-ca" pod "85f73d79-17ec-4ba0-a85c-08ba05fd9af6" (UID: "85f73d79-17ec-4ba0-a85c-08ba05fd9af6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:20.178297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.178262 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:20.178297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.178289 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7gm\" (UniqueName: \"kubernetes.io/projected/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c-kube-api-access-gs7gm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:20.181537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.181514 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-kube-api-access-kgjq7" pod "85f73d79-17ec-4ba0-a85c-08ba05fd9af6" (UID: "85f73d79-17ec-4ba0-a85c-08ba05fd9af6"). InnerVolumeSpecName "kube-api-access-kgjq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:20.182554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.182527 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv"] Nov 02 14:45:20.189691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.189662 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-605541-29893845-cvjwv"] Nov 02 14:45:20.218170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.218140 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv"] Nov 02 14:45:20.223772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.223740 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893845-q85bv"] Nov 02 14:45:20.254499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.254469 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn"] Nov 02 14:45:20.259302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.259238 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893845-x9xtn"] Nov 02 14:45:20.279683 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.279650 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgjq7\" (UniqueName: \"kubernetes.io/projected/85f73d79-17ec-4ba0-a85c-08ba05fd9af6-kube-api-access-kgjq7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:20.739045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.738908 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" event={"ID":"91775710-4a1d-4dbb-ab4d-d3c771f4cad2","Type":"ContainerStarted","Data":"ca4f8270d33950cb665de76ab7f653ff930f9caa15042c8411e91916d44b322e"} Nov 02 14:45:20.742271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.742242 5591 generic.go:356] "Generic (PLEG): container finished" podID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerID="f4ce287ca578ee75ae9a32e79ae13aa77b1d239979a699257982f8c46aac5f3e" exitCode=60 Nov 02 14:45:20.742398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.742313 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" event={"ID":"b9ce129d-05f8-48db-82cb-fa836c2992a0","Type":"ContainerDied","Data":"f4ce287ca578ee75ae9a32e79ae13aa77b1d239979a699257982f8c46aac5f3e"} Nov 02 14:45:20.742398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.742368 5591 scope.go:122] "RemoveContainer" containerID="595ecef24865126c8f5a2c53608abf8da20d8414d54c378c9b86f66649e912b4" Nov 02 14:45:20.742755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.742736 5591 scope.go:122] "RemoveContainer" containerID="f4ce287ca578ee75ae9a32e79ae13aa77b1d239979a699257982f8c46aac5f3e" Nov 02 14:45:20.743179 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:20.743154 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l_odh-ai-gateway-infra(b9ce129d-05f8-48db-82cb-fa836c2992a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" Nov 02 14:45:20.743816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.743797 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" event={"ID":"b7d03007-f6a2-47d4-8742-4ca9d4d2f71c","Type":"ContainerDied","Data":"18efc21e32b8bd221750bee00a59a3d77ad49dcf879adafb70680f3f7bcf6b75"} Nov 02 14:45:20.743816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.743812 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw" Nov 02 14:45:20.748073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.748050 5591 generic.go:356] "Generic (PLEG): container finished" podID="d0947e44-e37f-4591-adfe-714233cdbd98" containerID="65f1da06b22141139022160c079d68504784b4e1392b7bc70cc8e55fce0a02eb" exitCode=60 Nov 02 14:45:20.748183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.748121 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" event={"ID":"d0947e44-e37f-4591-adfe-714233cdbd98","Type":"ContainerDied","Data":"65f1da06b22141139022160c079d68504784b4e1392b7bc70cc8e55fce0a02eb"} Nov 02 14:45:20.748582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.748559 5591 scope.go:122] "RemoveContainer" containerID="65f1da06b22141139022160c079d68504784b4e1392b7bc70cc8e55fce0a02eb" Nov 02 14:45:20.748943 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:20.748917 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6_odh-ai-gateway-infra(d0947e44-e37f-4591-adfe-714233cdbd98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" Nov 02 14:45:20.750554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.750528 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" event={"ID":"cd174ecd-1998-4f54-8588-9de900552feb","Type":"ContainerStarted","Data":"f46a7b3c8a6d9ccff938ec009bdae589a67bec2e1ae102415c58819150ac5b9e"} Nov 02 14:45:20.752628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.752604 5591 generic.go:356] "Generic (PLEG): container finished" podID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerID="f08b4de40e181c569a2b2d92c2518560f708dacdb647c139921903b0eafd190b" exitCode=60 Nov 02 14:45:20.752716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.752667 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" event={"ID":"e59d93eb-2fa4-4094-98df-b0f1d75e0c94","Type":"ContainerDied","Data":"f08b4de40e181c569a2b2d92c2518560f708dacdb647c139921903b0eafd190b"} Nov 02 14:45:20.753061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.753039 5591 scope.go:122] "RemoveContainer" containerID="f08b4de40e181c569a2b2d92c2518560f708dacdb647c139921903b0eafd190b" Nov 02 14:45:20.753419 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:20.753329 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p_odh-ai-gateway-infra(e59d93eb-2fa4-4094-98df-b0f1d75e0c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" Nov 02 14:45:20.754953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.754928 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" event={"ID":"70f48a2c-7440-4363-bfc0-a7ccafdbed80","Type":"ContainerStarted","Data":"9e99eeb5f324e98849310318c17218f372811e390e53f824d05b32f45146ca06"} Nov 02 14:45:20.757469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.757452 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" Nov 02 14:45:20.757579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.757454 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6" event={"ID":"85f73d79-17ec-4ba0-a85c-08ba05fd9af6","Type":"ContainerDied","Data":"c348a4417912c0b0ebccfb02a92296bd8a909c769673392f35a25e87b40895b5"} Nov 02 14:45:20.759147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.759125 5591 scope.go:122] "RemoveContainer" containerID="65d7820b67d70fff2c9ea8d8d194b8396258ebc987fde08c35d955c952542a00" Nov 02 14:45:20.769839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.769819 5591 scope.go:122] "RemoveContainer" containerID="af10f9ae3635ec6e923427706d687f4c697e2ea83381a6ba268cced461cfcfac" Nov 02 14:45:20.773160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.773137 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg"] Nov 02 14:45:20.784748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.784725 5591 scope.go:122] "RemoveContainer" containerID="90aea3f8bbbf3aa3701d164df72d176a1415c4519006634ebc6c7a82f0f2ad45" Nov 02 14:45:20.800495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.800467 5591 scope.go:122] "RemoveContainer" containerID="5967922dfefae38b19fc87519f36806336a37cecdb27445ceea6e7cb3d8d1b37" Nov 02 14:45:20.809035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.808728 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p"] Nov 02 14:45:20.927081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.926447 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:20.929027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.929000 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l"] Nov 02 14:45:20.942267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.942238 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6"] Nov 02 14:45:20.955031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.948447 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893845-v6cc6"] Nov 02 14:45:20.955031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.954618 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6"] Nov 02 14:45:20.969939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.969912 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw"] Nov 02 14:45:20.975330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:20.975306 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893845-9t2kw"] Nov 02 14:45:21.259261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.259197 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:21.308162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.308130 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62221ac5-75fe-4999-9df5-2952cd3dee73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62221ac5-75fe-4999-9df5-2952cd3dee73-maas-api-service-ca\") pod \"62221ac5-75fe-4999-9df5-2952cd3dee73\" (UID: \"62221ac5-75fe-4999-9df5-2952cd3dee73\") " Nov 02 14:45:21.308376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.308212 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62221ac5-75fe-4999-9df5-2952cd3dee73-kube-api-access-g9t6w\" (UniqueName: \"kubernetes.io/projected/62221ac5-75fe-4999-9df5-2952cd3dee73-kube-api-access-g9t6w\") pod \"62221ac5-75fe-4999-9df5-2952cd3dee73\" (UID: \"62221ac5-75fe-4999-9df5-2952cd3dee73\") " Nov 02 14:45:21.309601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.309571 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62221ac5-75fe-4999-9df5-2952cd3dee73-maas-api-service-ca" pod "62221ac5-75fe-4999-9df5-2952cd3dee73" (UID: "62221ac5-75fe-4999-9df5-2952cd3dee73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:21.311395 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.311361 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62221ac5-75fe-4999-9df5-2952cd3dee73-kube-api-access-g9t6w" pod "62221ac5-75fe-4999-9df5-2952cd3dee73" (UID: "62221ac5-75fe-4999-9df5-2952cd3dee73"). InnerVolumeSpecName "kube-api-access-g9t6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:21.410509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.410476 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62221ac5-75fe-4999-9df5-2952cd3dee73-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:21.410509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.410508 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9t6w\" (UniqueName: \"kubernetes.io/projected/62221ac5-75fe-4999-9df5-2952cd3dee73-kube-api-access-g9t6w\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:21.433569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.433534 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" path="/var/lib/kubelet/pods/0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8/volumes" Nov 02 14:45:21.434199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.434153 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" path="/var/lib/kubelet/pods/85f73d79-17ec-4ba0-a85c-08ba05fd9af6/volumes" Nov 02 14:45:21.434629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.434612 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96800fad-b801-4f01-91db-db678d597bf1" path="/var/lib/kubelet/pods/96800fad-b801-4f01-91db-db678d597bf1/volumes" Nov 02 14:45:21.435353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.435333 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" path="/var/lib/kubelet/pods/b7d03007-f6a2-47d4-8742-4ca9d4d2f71c/volumes" Nov 02 14:45:21.435781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.435766 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" path="/var/lib/kubelet/pods/d6982912-1475-49a1-a605-2db8b5bc7687/volumes" Nov 02 14:45:21.436218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.436202 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" path="/var/lib/kubelet/pods/df7a7fb7-de79-44d6-ac70-753b949d5a12/volumes" Nov 02 14:45:21.771737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.771699 5591 generic.go:356] "Generic (PLEG): container finished" podID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerID="ca4f8270d33950cb665de76ab7f653ff930f9caa15042c8411e91916d44b322e" exitCode=60 Nov 02 14:45:21.772249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.771769 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" event={"ID":"91775710-4a1d-4dbb-ab4d-d3c771f4cad2","Type":"ContainerDied","Data":"ca4f8270d33950cb665de76ab7f653ff930f9caa15042c8411e91916d44b322e"} Nov 02 14:45:21.772249 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.771830 5591 scope.go:122] "RemoveContainer" containerID="33556771fc50391db0ca2463acb4741e9438f9f5f167beeeb0506427d7e8a597" Nov 02 14:45:21.772400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.772283 5591 scope.go:122] "RemoveContainer" containerID="ca4f8270d33950cb665de76ab7f653ff930f9caa15042c8411e91916d44b322e" Nov 02 14:45:21.772726 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:21.772703 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc_odh-ai-gateway-infra(91775710-4a1d-4dbb-ab4d-d3c771f4cad2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" Nov 02 14:45:21.779064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.779039 5591 generic.go:356] "Generic (PLEG): container finished" podID="cd174ecd-1998-4f54-8588-9de900552feb" containerID="f46a7b3c8a6d9ccff938ec009bdae589a67bec2e1ae102415c58819150ac5b9e" exitCode=60 Nov 02 14:45:21.779182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.779122 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" event={"ID":"cd174ecd-1998-4f54-8588-9de900552feb","Type":"ContainerDied","Data":"f46a7b3c8a6d9ccff938ec009bdae589a67bec2e1ae102415c58819150ac5b9e"} Nov 02 14:45:21.779597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.779549 5591 scope.go:122] "RemoveContainer" containerID="f46a7b3c8a6d9ccff938ec009bdae589a67bec2e1ae102415c58819150ac5b9e" Nov 02 14:45:21.780067 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:21.779978 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42_odh-ai-gateway-infra(cd174ecd-1998-4f54-8588-9de900552feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" podUID="cd174ecd-1998-4f54-8588-9de900552feb" Nov 02 14:45:21.787611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.787582 5591 generic.go:356] "Generic (PLEG): container finished" podID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerID="9e99eeb5f324e98849310318c17218f372811e390e53f824d05b32f45146ca06" exitCode=60 Nov 02 14:45:21.787747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.787725 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" event={"ID":"70f48a2c-7440-4363-bfc0-a7ccafdbed80","Type":"ContainerDied","Data":"9e99eeb5f324e98849310318c17218f372811e390e53f824d05b32f45146ca06"} Nov 02 14:45:21.788244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.788223 5591 scope.go:122] "RemoveContainer" containerID="9e99eeb5f324e98849310318c17218f372811e390e53f824d05b32f45146ca06" Nov 02 14:45:21.788680 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:21.788653 5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl_odh-ai-gateway-infra(70f48a2c-7440-4363-bfc0-a7ccafdbed80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" Nov 02 14:45:21.791534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.791504 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" event={"ID":"62221ac5-75fe-4999-9df5-2952cd3dee73","Type":"ContainerDied","Data":"f34d4faa1ca71a46a67cc3257433d7594f06038a3c1cafd995b363b6b3143539"} Nov 02 14:45:21.791635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.791595 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg" Nov 02 14:45:21.792367 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.792351 5591 scope.go:122] "RemoveContainer" containerID="d65ae15a04f39bd29ce74aa3ace357ffd42041283060cb898aece01517af1c72" Nov 02 14:45:21.814159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.814131 5591 scope.go:122] "RemoveContainer" containerID="fb4c9f735e675eb74fa032d5d705b7bf2b0b2e07f429636c707267036ab26a54" Nov 02 14:45:21.844132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.842372 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc"] Nov 02 14:45:21.844132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:21.843218 5591 scope.go:122] "RemoveContainer" containerID="cd21cd52d09cdbadbd3a4919d6ab2e233b11f8848750e3b6375fb5bc66a73e1f" Nov 02 14:45:22.002776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.002717 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:22.016211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.016179 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg"] Nov 02 14:45:22.022918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.022856 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893845-x5rqg"] Nov 02 14:45:22.028688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.028658 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0947e44-e37f-4591-adfe-714233cdbd98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0947e44-e37f-4591-adfe-714233cdbd98-maas-api-service-ca\") pod \"d0947e44-e37f-4591-adfe-714233cdbd98\" (UID: \"d0947e44-e37f-4591-adfe-714233cdbd98\") " Nov 02 14:45:22.028819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.028757 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0947e44-e37f-4591-adfe-714233cdbd98-kube-api-access-956pg\" (UniqueName: \"kubernetes.io/projected/d0947e44-e37f-4591-adfe-714233cdbd98-kube-api-access-956pg\") pod \"d0947e44-e37f-4591-adfe-714233cdbd98\" (UID: \"d0947e44-e37f-4591-adfe-714233cdbd98\") " Nov 02 14:45:22.029444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.029415 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0947e44-e37f-4591-adfe-714233cdbd98-maas-api-service-ca" pod "d0947e44-e37f-4591-adfe-714233cdbd98" (UID: "d0947e44-e37f-4591-adfe-714233cdbd98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:22.031053 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.031026 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl"] Nov 02 14:45:22.033195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.033148 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0947e44-e37f-4591-adfe-714233cdbd98-kube-api-access-956pg" pod "d0947e44-e37f-4591-adfe-714233cdbd98" (UID: "d0947e44-e37f-4591-adfe-714233cdbd98"). InnerVolumeSpecName "kube-api-access-956pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:22.079136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.079103 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:22.083396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.083364 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:22.129930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.129894 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-kube-api-access-fl6pd\" (UniqueName: \"kubernetes.io/projected/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-kube-api-access-fl6pd\") pod \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\" (UID: \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\") " Nov 02 14:45:22.130139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.129953 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ce129d-05f8-48db-82cb-fa836c2992a0-kube-api-access-v68kn\" (UniqueName: \"kubernetes.io/projected/b9ce129d-05f8-48db-82cb-fa836c2992a0-kube-api-access-v68kn\") pod \"b9ce129d-05f8-48db-82cb-fa836c2992a0\" (UID: \"b9ce129d-05f8-48db-82cb-fa836c2992a0\") " Nov 02 14:45:22.130139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.130126 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ce129d-05f8-48db-82cb-fa836c2992a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce129d-05f8-48db-82cb-fa836c2992a0-maas-api-service-ca\") pod \"b9ce129d-05f8-48db-82cb-fa836c2992a0\" (UID: \"b9ce129d-05f8-48db-82cb-fa836c2992a0\") " Nov 02 14:45:22.130232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.130148 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-maas-api-service-ca\") pod \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\" (UID: \"e59d93eb-2fa4-4094-98df-b0f1d75e0c94\") " Nov 02 14:45:22.130500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.130474 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ce129d-05f8-48db-82cb-fa836c2992a0-maas-api-service-ca" pod "b9ce129d-05f8-48db-82cb-fa836c2992a0" (UID: "b9ce129d-05f8-48db-82cb-fa836c2992a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:22.130636 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.130560 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-maas-api-service-ca" pod "e59d93eb-2fa4-4094-98df-b0f1d75e0c94" (UID: "e59d93eb-2fa4-4094-98df-b0f1d75e0c94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:22.130794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.130775 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ce129d-05f8-48db-82cb-fa836c2992a0-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:22.130863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.130794 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:22.130863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.130805 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0947e44-e37f-4591-adfe-714233cdbd98-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:22.130863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.130815 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-956pg\" (UniqueName: \"kubernetes.io/projected/d0947e44-e37f-4591-adfe-714233cdbd98-kube-api-access-956pg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:22.132229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.132204 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ce129d-05f8-48db-82cb-fa836c2992a0-kube-api-access-v68kn" pod "b9ce129d-05f8-48db-82cb-fa836c2992a0" (UID: "b9ce129d-05f8-48db-82cb-fa836c2992a0"). InnerVolumeSpecName "kube-api-access-v68kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:22.132334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.132233 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-kube-api-access-fl6pd" pod "e59d93eb-2fa4-4094-98df-b0f1d75e0c94" (UID: "e59d93eb-2fa4-4094-98df-b0f1d75e0c94"). InnerVolumeSpecName "kube-api-access-fl6pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:22.202206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.202172 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42"] Nov 02 14:45:22.232393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.232357 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl6pd\" (UniqueName: \"kubernetes.io/projected/e59d93eb-2fa4-4094-98df-b0f1d75e0c94-kube-api-access-fl6pd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:22.232393 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.232388 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v68kn\" (UniqueName: \"kubernetes.io/projected/b9ce129d-05f8-48db-82cb-fa836c2992a0-kube-api-access-v68kn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:22.788055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.782220 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:22.826021 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.825957 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" event={"ID":"b9ce129d-05f8-48db-82cb-fa836c2992a0","Type":"ContainerDied","Data":"0ad74f3059783bcf11cb8613d6684808bf0e953bd0963d18d5844a8ad719b051"} Nov 02 14:45:22.826137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.826040 5591 scope.go:122] "RemoveContainer" containerID="f4ce287ca578ee75ae9a32e79ae13aa77b1d239979a699257982f8c46aac5f3e" Nov 02 14:45:22.826454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.826265 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l" Nov 02 14:45:22.828091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.828056 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" event={"ID":"d0947e44-e37f-4591-adfe-714233cdbd98","Type":"ContainerDied","Data":"26677b5e29acc9e4458b271f7a8af54e90be7c044535611d90bace010c913289"} Nov 02 14:45:22.828204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.828169 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6" Nov 02 14:45:22.849691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.849604 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" event={"ID":"e59d93eb-2fa4-4094-98df-b0f1d75e0c94","Type":"ContainerDied","Data":"fc9895329272f424b53f39965c83dad090605ab51c3ff19006ac2b8b39b11cbc"} Nov 02 14:45:22.849691 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.849611 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p" Nov 02 14:45:22.850488 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.850127 5591 scope.go:122] "RemoveContainer" containerID="65f1da06b22141139022160c079d68504784b4e1392b7bc70cc8e55fce0a02eb" Nov 02 14:45:22.863879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.863858 5591 scope.go:122] "RemoveContainer" containerID="f08b4de40e181c569a2b2d92c2518560f708dacdb647c139921903b0eafd190b" Nov 02 14:45:22.959126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.959058 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6"] Nov 02 14:45:22.966072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.966035 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893845-bc4d6"] Nov 02 14:45:22.984282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:22.984203 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:23.095686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.095621 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p"] Nov 02 14:45:23.110551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.110517 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:23.118253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.118221 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893845-p2q2p"] Nov 02 14:45:23.136012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.135960 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l"] Nov 02 14:45:23.147494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.147158 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-66bc6d-29893845-pl42l"] Nov 02 14:45:23.160751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.160721 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-kube-api-access-pw67f\" (UniqueName: \"kubernetes.io/projected/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-kube-api-access-pw67f\") pod \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\" (UID: \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\") " Nov 02 14:45:23.161165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.161140 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-maas-api-service-ca\") pod \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\" (UID: \"91775710-4a1d-4dbb-ab4d-d3c771f4cad2\") " Nov 02 14:45:23.161712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.161590 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-maas-api-service-ca" pod "91775710-4a1d-4dbb-ab4d-d3c771f4cad2" (UID: "91775710-4a1d-4dbb-ab4d-d3c771f4cad2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:23.166129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.166105 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.168801 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.168718 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:23.172658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.172553 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-kube-api-access-pw67f" pod "91775710-4a1d-4dbb-ab4d-d3c771f4cad2" (UID: "91775710-4a1d-4dbb-ab4d-d3c771f4cad2"). InnerVolumeSpecName "kube-api-access-pw67f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:23.187947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.186755 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:23.265722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.265694 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:45:23.266933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.266908 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f48a2c-7440-4363-bfc0-a7ccafdbed80-kube-api-access-m8vwz\" (UniqueName: \"kubernetes.io/projected/70f48a2c-7440-4363-bfc0-a7ccafdbed80-kube-api-access-m8vwz\") pod \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\" (UID: \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\") " Nov 02 14:45:23.267121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.266959 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f48a2c-7440-4363-bfc0-a7ccafdbed80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f48a2c-7440-4363-bfc0-a7ccafdbed80-maas-api-service-ca\") pod \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\" (UID: \"70f48a2c-7440-4363-bfc0-a7ccafdbed80\") " Nov 02 14:45:23.267121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.267062 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd174ecd-1998-4f54-8588-9de900552feb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd174ecd-1998-4f54-8588-9de900552feb-maas-api-service-ca\") pod \"cd174ecd-1998-4f54-8588-9de900552feb\" (UID: \"cd174ecd-1998-4f54-8588-9de900552feb\") " Nov 02 14:45:23.267330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.267259 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd174ecd-1998-4f54-8588-9de900552feb-kube-api-access-tpkdb\" (UniqueName: \"kubernetes.io/projected/cd174ecd-1998-4f54-8588-9de900552feb-kube-api-access-tpkdb\") pod \"cd174ecd-1998-4f54-8588-9de900552feb\" (UID: \"cd174ecd-1998-4f54-8588-9de900552feb\") " Nov 02 14:45:23.267400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.267369 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f48a2c-7440-4363-bfc0-a7ccafdbed80-maas-api-service-ca" pod "70f48a2c-7440-4363-bfc0-a7ccafdbed80" (UID: "70f48a2c-7440-4363-bfc0-a7ccafdbed80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:23.267489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.267463 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd174ecd-1998-4f54-8588-9de900552feb-maas-api-service-ca" pod "cd174ecd-1998-4f54-8588-9de900552feb" (UID: "cd174ecd-1998-4f54-8588-9de900552feb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:23.268126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.268104 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd174ecd-1998-4f54-8588-9de900552feb-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.268225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.268129 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw67f\" (UniqueName: \"kubernetes.io/projected/91775710-4a1d-4dbb-ab4d-d3c771f4cad2-kube-api-access-pw67f\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.268225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.268146 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f48a2c-7440-4363-bfc0-a7ccafdbed80-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.269526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.269500 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f48a2c-7440-4363-bfc0-a7ccafdbed80-kube-api-access-m8vwz" pod "70f48a2c-7440-4363-bfc0-a7ccafdbed80" (UID: "70f48a2c-7440-4363-bfc0-a7ccafdbed80"). InnerVolumeSpecName "kube-api-access-m8vwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:23.269612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.269570 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd174ecd-1998-4f54-8588-9de900552feb-kube-api-access-tpkdb" pod "cd174ecd-1998-4f54-8588-9de900552feb" (UID: "cd174ecd-1998-4f54-8588-9de900552feb"). InnerVolumeSpecName "kube-api-access-tpkdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:23.369338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369246 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istiod-ca-cert\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.369338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369309 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-socket\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.369338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369335 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-certs\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.369630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369533 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-data\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.369630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369584 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-podinfo\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.369731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369656 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-kube-api-access-tp6qd\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-kube-api-access-tp6qd\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.369731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369654 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-socket" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:23.369731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369685 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istiod-ca-cert" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:23.369883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369794 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-token\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-token\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.369883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369844 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-credential-socket\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.370049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.369929 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-envoy\") pod \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\" (UID: \"0d71f880-d4ea-4d6d-9505-bccb5c82253b\") " Nov 02 14:45:23.370199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370176 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-data" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:23.370300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370281 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-credential-socket" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:23.370400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370379 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-certs" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:23.370790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370764 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpkdb\" (UniqueName: \"kubernetes.io/projected/cd174ecd-1998-4f54-8588-9de900552feb-kube-api-access-tpkdb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.370790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370795 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.371070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370810 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.371070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370826 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.371070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370840 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.371070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370854 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vwz\" (UniqueName: \"kubernetes.io/projected/70f48a2c-7440-4363-bfc0-a7ccafdbed80-kube-api-access-m8vwz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.371070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.370868 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.372242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.372199 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-podinfo" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:45:23.372479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.372460 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-kube-api-access-tp6qd" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "kube-api-access-tp6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:23.372523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.372495 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-token" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:23.372835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.372815 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-envoy" pod "0d71f880-d4ea-4d6d-9505-bccb5c82253b" (UID: "0d71f880-d4ea-4d6d-9505-bccb5c82253b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:23.437079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.437044 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" path="/var/lib/kubelet/pods/62221ac5-75fe-4999-9df5-2952cd3dee73/volumes" Nov 02 14:45:23.437580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.437565 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" path="/var/lib/kubelet/pods/b9ce129d-05f8-48db-82cb-fa836c2992a0/volumes" Nov 02 14:45:23.437977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.437963 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" path="/var/lib/kubelet/pods/d0947e44-e37f-4591-adfe-714233cdbd98/volumes" Nov 02 14:45:23.438701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.438684 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" path="/var/lib/kubelet/pods/e59d93eb-2fa4-4094-98df-b0f1d75e0c94/volumes" Nov 02 14:45:23.472951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.472924 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.472951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.472946 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp6qd\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-kube-api-access-tp6qd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.473147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.472963 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.473147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.472972 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d71f880-d4ea-4d6d-9505-bccb5c82253b-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:23.713329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.709489 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:23.864742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.864706 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" Nov 02 14:45:23.864742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.864705 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc" event={"ID":"91775710-4a1d-4dbb-ab4d-d3c771f4cad2","Type":"ContainerDied","Data":"b2875f0ea15901e47f60e6ab897d51753ff78df694c419d4fb8632d849f1cf79"} Nov 02 14:45:23.865348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.864826 5591 scope.go:122] "RemoveContainer" containerID="ca4f8270d33950cb665de76ab7f653ff930f9caa15042c8411e91916d44b322e" Nov 02 14:45:23.869613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.869584 5591 generic.go:356] "Generic (PLEG): container finished" podID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerID="c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851" exitCode=0 Nov 02 14:45:23.869756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.869657 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" Nov 02 14:45:23.869756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.869678 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" event={"ID":"0d71f880-d4ea-4d6d-9505-bccb5c82253b","Type":"ContainerDied","Data":"c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851"} Nov 02 14:45:23.869756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.869744 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2" event={"ID":"0d71f880-d4ea-4d6d-9505-bccb5c82253b","Type":"ContainerDied","Data":"58de4b51797b88a71a8b23c19d0ac654364754b024a47de58bfc6e138843ada4"} Nov 02 14:45:23.873938 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.873912 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" Nov 02 14:45:23.873938 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.873932 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42" event={"ID":"cd174ecd-1998-4f54-8588-9de900552feb","Type":"ContainerDied","Data":"a5addec6796896aca71c5978c80ee87f6bb0d075469772a55031f66bbcc44203"} Nov 02 14:45:23.876834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.876801 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" event={"ID":"70f48a2c-7440-4363-bfc0-a7ccafdbed80","Type":"ContainerDied","Data":"15952698a75779c795f4e9db20a7af54b6d03bcfffa4cefa00e54bdaec7b1ae4"} Nov 02 14:45:23.876834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.876829 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl" Nov 02 14:45:23.881385 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.881363 5591 scope.go:122] "RemoveContainer" containerID="c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851" Nov 02 14:45:23.895319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.895294 5591 scope.go:122] "RemoveContainer" containerID="75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5" Nov 02 14:45:23.907280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.907255 5591 scope.go:122] "RemoveContainer" containerID="c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851" Nov 02 14:45:23.907676 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:23.907651 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851\": container with ID starting with c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851 not found: ID does not exist" containerID="c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851" Nov 02 14:45:23.907788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.907681 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851"} err="failed to get container status \"c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851\": rpc error: code = NotFound desc = could not find container \"c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851\": container with ID starting with c7b9b4b46b87ae0371f974a4f7e000b68c15c2c8b2b5ef9d83b9205c0f33d851 not found: ID does not exist" Nov 02 14:45:23.907788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.907700 5591 scope.go:122] "RemoveContainer" containerID="75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5" Nov 02 14:45:23.908002 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:23.907965 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5\": container with ID starting with 75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5 not found: ID does not exist" containerID="75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5" Nov 02 14:45:23.908096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.908006 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5"} err="failed to get container status \"75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5\": rpc error: code = NotFound desc = could not find container \"75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5\": container with ID starting with 75b01d6ef217022433751966d89970afe0ceab5ef2d0dfb80c7919909fe04eb5 not found: ID does not exist" Nov 02 14:45:23.908096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.908021 5591 scope.go:122] "RemoveContainer" containerID="f46a7b3c8a6d9ccff938ec009bdae589a67bec2e1ae102415c58819150ac5b9e" Nov 02 14:45:23.918042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.918023 5591 scope.go:122] "RemoveContainer" containerID="9e99eeb5f324e98849310318c17218f372811e390e53f824d05b32f45146ca06" Nov 02 14:45:23.944119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.944092 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2"] Nov 02 14:45:23.956720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.956691 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-msbk2"] Nov 02 14:45:23.987448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.987366 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl"] Nov 02 14:45:23.992215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:23.992189 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893845-nxpsl"] Nov 02 14:45:24.033115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:24.033080 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc"] Nov 02 14:45:24.064707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:24.064662 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893845-2h7vc"] Nov 02 14:45:24.100450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:24.100417 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42"] Nov 02 14:45:24.108658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:24.108626 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893845-pvv42"] Nov 02 14:45:24.159815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:24.159776 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:24.761355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:24.761319 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:25.035574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:25.035494 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:25.436675 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:25.436639 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" path="/var/lib/kubelet/pods/0d71f880-d4ea-4d6d-9505-bccb5c82253b/volumes" Nov 02 14:45:25.437277 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:25.437252 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" path="/var/lib/kubelet/pods/70f48a2c-7440-4363-bfc0-a7ccafdbed80/volumes" Nov 02 14:45:25.437709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:25.437686 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" path="/var/lib/kubelet/pods/91775710-4a1d-4dbb-ab4d-d3c771f4cad2/volumes" Nov 02 14:45:25.438453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:25.438432 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd174ecd-1998-4f54-8588-9de900552feb" path="/var/lib/kubelet/pods/cd174ecd-1998-4f54-8588-9de900552feb/volumes" Nov 02 14:45:26.686462 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:26.664180 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:26.901401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:26.901351 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:26.945944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:26.945845 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m"] Nov 02 14:45:26.978528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:26.978498 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2"] Nov 02 14:45:27.128451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:27.128418 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:27.314321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:27.314186 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:27.480407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:27.480371 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:28.407271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:28.407237 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:30.081620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:30.081582 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:31.075897 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:31.075853 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:45:31.221815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:31.221783 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl"] Nov 02 14:45:31.380756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:31.380681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:31.542043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:31.542011 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:31.710844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:31.710812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:31.927009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:31.926705 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:31.949010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:31.946844 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" podUID="54e490c6-5154-4333-be08-3bf46465f4de" containerName="payload-processing" containerID="cri-o://8c9eb39407805985674b206cabf468e95c94726509bcccca8486fa66844d7499" gracePeriod=25 Nov 02 14:45:31.979831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:31.979688 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" podUID="64bb2eb4-559d-4b7c-8833-a76ba98da86e" containerName="payload-pre-processing" containerID="cri-o://8cbd1d7285fe5fc4f0d985fe3e05d44bcb422e0a3e9ac9698fdd4957821aeb7f" gracePeriod=25 Nov 02 14:45:33.026247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:33.026209 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:33.467624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:33.464087 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:35.028185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:35.028144 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:35.441702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:35.441661 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv"] Nov 02 14:45:35.441951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:35.441919 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" containerID="cri-o://c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81" gracePeriod=30 Nov 02 14:45:36.222376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:36.222332 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" podUID="825bfd14-a3a6-4436-9286-3f3ea1f3fd49" containerName="payload-pre-processing" containerID="cri-o://cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879" gracePeriod=25 Nov 02 14:45:36.702542 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:36.702504 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:36.992718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:36.992629 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:45:36.992718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:36.992696 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:45:38.024910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:38.024873 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:38.467099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:38.467055 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:39.438649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:39.438614 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" podUID="825bfd14-a3a6-4436-9286-3f3ea1f3fd49" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:45:40.231008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.228530 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:40.736080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.736049 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 14:45:40.904100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904066 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-envoy\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904110 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-socket\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904222 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-token\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-token\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904246 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-credential-socket\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904266 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2fb505-7f16-449b-9b80-ae516cf2bba7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2fb505-7f16-449b-9b80-ae516cf2bba7-istiod-ca-cert\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904292 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-podinfo\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904393 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-certs\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904442 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-socket" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:40.904507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904448 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-data\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904552 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-kube-api-access-qsdp4\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-kube-api-access-qsdp4\") pod \"be2fb505-7f16-449b-9b80-ae516cf2bba7\" (UID: \"be2fb505-7f16-449b-9b80-ae516cf2bba7\") " Nov 02 14:45:40.904722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904657 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-data" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:40.904722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904680 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2fb505-7f16-449b-9b80-ae516cf2bba7-istiod-ca-cert" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:40.904889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904853 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-certs" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:40.904889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.904874 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-credential-socket" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:40.905571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.905477 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:40.905571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.905505 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:40.905571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.905519 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:40.905571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.905532 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:40.905571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.905545 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2fb505-7f16-449b-9b80-ae516cf2bba7-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:40.906755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.906717 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-envoy" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:45:40.906755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.906737 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-podinfo" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:45:40.906903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.906774 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-token" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:40.907163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:40.907148 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-kube-api-access-qsdp4" pod "be2fb505-7f16-449b-9b80-ae516cf2bba7" (UID: "be2fb505-7f16-449b-9b80-ae516cf2bba7"). InnerVolumeSpecName "kube-api-access-qsdp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:41.007477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.007443 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:41.007477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.007474 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:41.007477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.007484 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsdp4\" (UniqueName: \"kubernetes.io/projected/be2fb505-7f16-449b-9b80-ae516cf2bba7-kube-api-access-qsdp4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:41.007720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.007493 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2fb505-7f16-449b-9b80-ae516cf2bba7-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:41.041626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.041588 5591 generic.go:356] "Generic (PLEG): container finished" podID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerID="c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81" exitCode=0 Nov 02 14:45:41.041816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.041637 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" Nov 02 14:45:41.041816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.041681 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" event={"ID":"be2fb505-7f16-449b-9b80-ae516cf2bba7","Type":"ContainerDied","Data":"c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81"} Nov 02 14:45:41.041816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.041751 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv" event={"ID":"be2fb505-7f16-449b-9b80-ae516cf2bba7","Type":"ContainerDied","Data":"eb56b4679fcb8b1eeffd20f9886422e49e5f6ffbe666ad73ef9937b1f179a297"} Nov 02 14:45:41.041816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.041783 5591 scope.go:122] "RemoveContainer" containerID="c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81" Nov 02 14:45:41.054358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.054331 5591 scope.go:122] "RemoveContainer" containerID="c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81" Nov 02 14:45:41.054718 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:41.054698 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81\": container with ID starting with c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81 not found: ID does not exist" containerID="c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81" Nov 02 14:45:41.054768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.054728 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81"} err="failed to get container status \"c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81\": rpc error: code = NotFound desc = could not find container \"c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81\": container with ID starting with c74efa1f2e221692a671b1b7b7b5636cdf3eca475f9e98394500a03076b33d81 not found: ID does not exist" Nov 02 14:45:41.098637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.098597 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv"] Nov 02 14:45:41.111860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.111830 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-jn4hv"] Nov 02 14:45:41.434859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.434820 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" path="/var/lib/kubelet/pods/be2fb505-7f16-449b-9b80-ae516cf2bba7/volumes" Nov 02 14:45:41.909641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.909602 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" podUID="54e490c6-5154-4333-be08-3bf46465f4de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:45:41.910069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.909610 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" podUID="64bb2eb4-559d-4b7c-8833-a76ba98da86e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:45:41.935507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:41.935467 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:43.632118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:43.632077 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:45.334851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:45.334818 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:45.543668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:45.543629 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g"] Nov 02 14:45:45.670346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:45.670269 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:47.049873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:47.049837 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:48.705835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:48.705800 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:49.437959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:49.437919 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" podUID="825bfd14-a3a6-4436-9286-3f3ea1f3fd49" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:45:50.276016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:50.274293 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:51.909958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:51.909914 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" podUID="64bb2eb4-559d-4b7c-8833-a76ba98da86e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:45:51.910401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:51.909931 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" podUID="54e490c6-5154-4333-be08-3bf46465f4de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:45:51.953024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:51.952837 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:52.136033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.135964 5591 generic.go:356] "Generic (PLEG): container finished" podID="64bb2eb4-559d-4b7c-8833-a76ba98da86e" containerID="8cbd1d7285fe5fc4f0d985fe3e05d44bcb422e0a3e9ac9698fdd4957821aeb7f" exitCode=0 Nov 02 14:45:52.136228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.136113 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" event={"ID":"64bb2eb4-559d-4b7c-8833-a76ba98da86e","Type":"ContainerDied","Data":"8cbd1d7285fe5fc4f0d985fe3e05d44bcb422e0a3e9ac9698fdd4957821aeb7f"} Nov 02 14:45:52.138174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.138142 5591 generic.go:356] "Generic (PLEG): container finished" podID="54e490c6-5154-4333-be08-3bf46465f4de" containerID="8c9eb39407805985674b206cabf468e95c94726509bcccca8486fa66844d7499" exitCode=0 Nov 02 14:45:52.138291 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.138220 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" event={"ID":"54e490c6-5154-4333-be08-3bf46465f4de","Type":"ContainerDied","Data":"8c9eb39407805985674b206cabf468e95c94726509bcccca8486fa66844d7499"} Nov 02 14:45:52.277973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.277940 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:45:52.309871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.309844 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:45:52.421624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.421592 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e490c6-5154-4333-be08-3bf46465f4de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e490c6-5154-4333-be08-3bf46465f4de-plugins-config-volume\") pod \"54e490c6-5154-4333-be08-3bf46465f4de\" (UID: \"54e490c6-5154-4333-be08-3bf46465f4de\") " Nov 02 14:45:52.421836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.421815 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e490c6-5154-4333-be08-3bf46465f4de-kube-api-access-4g7q2\" (UniqueName: \"kubernetes.io/projected/54e490c6-5154-4333-be08-3bf46465f4de-kube-api-access-4g7q2\") pod \"54e490c6-5154-4333-be08-3bf46465f4de\" (UID: \"54e490c6-5154-4333-be08-3bf46465f4de\") " Nov 02 14:45:52.421916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.421899 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bb2eb4-559d-4b7c-8833-a76ba98da86e-kube-api-access-jcxhv\" (UniqueName: \"kubernetes.io/projected/64bb2eb4-559d-4b7c-8833-a76ba98da86e-kube-api-access-jcxhv\") pod \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\" (UID: \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\") " Nov 02 14:45:52.422037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.422015 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e490c6-5154-4333-be08-3bf46465f4de-plugins-config-volume" pod "54e490c6-5154-4333-be08-3bf46465f4de" (UID: "54e490c6-5154-4333-be08-3bf46465f4de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:52.422122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.422104 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bb2eb4-559d-4b7c-8833-a76ba98da86e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bb2eb4-559d-4b7c-8833-a76ba98da86e-plugins-config-volume\") pod \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\" (UID: \"64bb2eb4-559d-4b7c-8833-a76ba98da86e\") " Nov 02 14:45:52.422508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.422486 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bb2eb4-559d-4b7c-8833-a76ba98da86e-plugins-config-volume" pod "64bb2eb4-559d-4b7c-8833-a76ba98da86e" (UID: "64bb2eb4-559d-4b7c-8833-a76ba98da86e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:52.422810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.422789 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64bb2eb4-559d-4b7c-8833-a76ba98da86e-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:52.422893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.422817 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e490c6-5154-4333-be08-3bf46465f4de-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:52.424214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.424194 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e490c6-5154-4333-be08-3bf46465f4de-kube-api-access-4g7q2" pod "54e490c6-5154-4333-be08-3bf46465f4de" (UID: "54e490c6-5154-4333-be08-3bf46465f4de"). InnerVolumeSpecName "kube-api-access-4g7q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:52.424214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.424221 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bb2eb4-559d-4b7c-8833-a76ba98da86e-kube-api-access-jcxhv" pod "64bb2eb4-559d-4b7c-8833-a76ba98da86e" (UID: "64bb2eb4-559d-4b7c-8833-a76ba98da86e"). InnerVolumeSpecName "kube-api-access-jcxhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:52.524531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.524449 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g7q2\" (UniqueName: \"kubernetes.io/projected/54e490c6-5154-4333-be08-3bf46465f4de-kube-api-access-4g7q2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:52.524531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:52.524478 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxhv\" (UniqueName: \"kubernetes.io/projected/64bb2eb4-559d-4b7c-8833-a76ba98da86e-kube-api-access-jcxhv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:53.147847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.147813 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" Nov 02 14:45:53.147847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.147824 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2" event={"ID":"64bb2eb4-559d-4b7c-8833-a76ba98da86e","Type":"ContainerDied","Data":"26e671963ee93bb16c0dbbf091d4a1f4eff28c25b87067ee7982fbdc5bee043b"} Nov 02 14:45:53.148452 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.147900 5591 scope.go:122] "RemoveContainer" containerID="8cbd1d7285fe5fc4f0d985fe3e05d44bcb422e0a3e9ac9698fdd4957821aeb7f" Nov 02 14:45:53.149765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.149748 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" Nov 02 14:45:53.149869 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.149801 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m" event={"ID":"54e490c6-5154-4333-be08-3bf46465f4de","Type":"ContainerDied","Data":"94946849de04c724a790fc09af6d628ac4d4835820a4568032fa196b347329f3"} Nov 02 14:45:53.160227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.160210 5591 scope.go:122] "RemoveContainer" containerID="8c9eb39407805985674b206cabf468e95c94726509bcccca8486fa66844d7499" Nov 02 14:45:53.202879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.202850 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2"] Nov 02 14:45:53.208410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.208383 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d39fa6-6544f94499-n24s2"] Nov 02 14:45:53.243716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.243643 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m"] Nov 02 14:45:53.248334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.248310 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d39fa6-66f4b586c7-ccr2m"] Nov 02 14:45:53.432818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.432788 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e490c6-5154-4333-be08-3bf46465f4de" path="/var/lib/kubelet/pods/54e490c6-5154-4333-be08-3bf46465f4de/volumes" Nov 02 14:45:53.433300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.433280 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bb2eb4-559d-4b7c-8833-a76ba98da86e" path="/var/lib/kubelet/pods/64bb2eb4-559d-4b7c-8833-a76ba98da86e/volumes" Nov 02 14:45:53.517541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:53.510389 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:55.106017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:55.103287 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:56.512390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:56.512359 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:45:56.584699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:56.584658 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-kube-api-access-wtttf\" (UniqueName: \"kubernetes.io/projected/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-kube-api-access-wtttf\") pod \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\" (UID: \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\") " Nov 02 14:45:56.584897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:56.584852 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-plugins-config-volume\") pod \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\" (UID: \"825bfd14-a3a6-4436-9286-3f3ea1f3fd49\") " Nov 02 14:45:56.585320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:56.585283 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-plugins-config-volume" pod "825bfd14-a3a6-4436-9286-3f3ea1f3fd49" (UID: "825bfd14-a3a6-4436-9286-3f3ea1f3fd49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:45:56.585646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:56.585624 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:56.586894 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:56.586870 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-kube-api-access-wtttf" pod "825bfd14-a3a6-4436-9286-3f3ea1f3fd49" (UID: "825bfd14-a3a6-4436-9286-3f3ea1f3fd49"). InnerVolumeSpecName "kube-api-access-wtttf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:45:56.687413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:56.687366 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtttf\" (UniqueName: \"kubernetes.io/projected/825bfd14-a3a6-4436-9286-3f3ea1f3fd49-kube-api-access-wtttf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:45:56.903660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:56.903621 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:45:57.184704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.184623 5591 generic.go:356] "Generic (PLEG): container finished" podID="825bfd14-a3a6-4436-9286-3f3ea1f3fd49" containerID="cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879" exitCode=0 Nov 02 14:45:57.184704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.184671 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" Nov 02 14:45:57.184914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.184708 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" event={"ID":"825bfd14-a3a6-4436-9286-3f3ea1f3fd49","Type":"ContainerDied","Data":"cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879"} Nov 02 14:45:57.184914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.184793 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl" event={"ID":"825bfd14-a3a6-4436-9286-3f3ea1f3fd49","Type":"ContainerDied","Data":"e82900049c5c7a57c08c30ec7d611669595a3f7165a267bedec11884111d24fd"} Nov 02 14:45:57.184914 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.184821 5591 scope.go:122] "RemoveContainer" containerID="cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879" Nov 02 14:45:57.196585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.196560 5591 scope.go:122] "RemoveContainer" containerID="cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879" Nov 02 14:45:57.196861 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:45:57.196838 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879\": container with ID starting with cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879 not found: ID does not exist" containerID="cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879" Nov 02 14:45:57.196919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.196869 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879"} err="failed to get container status \"cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879\": rpc error: code = NotFound desc = could not find container \"cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879\": container with ID starting with cbb51a3db37bfa6409e9bfaf11b7143e33f39696670841acce2bc08086158879 not found: ID does not exist" Nov 02 14:45:57.239893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.239855 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl"] Nov 02 14:45:57.245812 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.245785 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e1136e-6d45d4ffdc-95hbl"] Nov 02 14:45:57.434628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:57.434594 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825bfd14-a3a6-4436-9286-3f3ea1f3fd49" path="/var/lib/kubelet/pods/825bfd14-a3a6-4436-9286-3f3ea1f3fd49/volumes" Nov 02 14:45:58.473955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:45:58.471369 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:00.177172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.177113 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:00.571555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.571458 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" podUID="70742c94-86ad-4a7f-8203-e22a7641366c" containerName="main" containerID="cri-o://d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae" gracePeriod=45 Nov 02 14:46:00.852042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.852012 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:46:00.967275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967237 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-home\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-home\") pod \"70742c94-86ad-4a7f-8203-e22a7641366c\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " Nov 02 14:46:00.967465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967359 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70742c94-86ad-4a7f-8203-e22a7641366c-tls-certs\" (UniqueName: \"kubernetes.io/secret/70742c94-86ad-4a7f-8203-e22a7641366c-tls-certs\") pod \"70742c94-86ad-4a7f-8203-e22a7641366c\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " Nov 02 14:46:00.967465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967410 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-tmp-dir\") pod \"70742c94-86ad-4a7f-8203-e22a7641366c\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " Nov 02 14:46:00.967465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967446 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70742c94-86ad-4a7f-8203-e22a7641366c-kube-api-access-gs2ds\" (UniqueName: \"kubernetes.io/projected/70742c94-86ad-4a7f-8203-e22a7641366c-kube-api-access-gs2ds\") pod \"70742c94-86ad-4a7f-8203-e22a7641366c\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " Nov 02 14:46:00.967661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967502 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-model-cache\") pod \"70742c94-86ad-4a7f-8203-e22a7641366c\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " Nov 02 14:46:00.967661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967543 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-home" pod "70742c94-86ad-4a7f-8203-e22a7641366c" (UID: "70742c94-86ad-4a7f-8203-e22a7641366c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:00.967661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967576 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-dshm\") pod \"70742c94-86ad-4a7f-8203-e22a7641366c\" (UID: \"70742c94-86ad-4a7f-8203-e22a7641366c\") " Nov 02 14:46:00.967823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967737 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-tmp-dir" pod "70742c94-86ad-4a7f-8203-e22a7641366c" (UID: "70742c94-86ad-4a7f-8203-e22a7641366c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:00.967890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.967827 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-model-cache" pod "70742c94-86ad-4a7f-8203-e22a7641366c" (UID: "70742c94-86ad-4a7f-8203-e22a7641366c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:00.968247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.968223 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:00.968247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.968247 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:00.968438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.968262 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:00.969717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.969688 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-dshm" pod "70742c94-86ad-4a7f-8203-e22a7641366c" (UID: "70742c94-86ad-4a7f-8203-e22a7641366c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:00.969830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.969765 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70742c94-86ad-4a7f-8203-e22a7641366c-tls-certs" pod "70742c94-86ad-4a7f-8203-e22a7641366c" (UID: "70742c94-86ad-4a7f-8203-e22a7641366c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:46:00.969830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:00.969792 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70742c94-86ad-4a7f-8203-e22a7641366c-kube-api-access-gs2ds" pod "70742c94-86ad-4a7f-8203-e22a7641366c" (UID: "70742c94-86ad-4a7f-8203-e22a7641366c"). InnerVolumeSpecName "kube-api-access-gs2ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:46:01.069448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.069412 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs2ds\" (UniqueName: \"kubernetes.io/projected/70742c94-86ad-4a7f-8203-e22a7641366c-kube-api-access-gs2ds\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:01.069448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.069442 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70742c94-86ad-4a7f-8203-e22a7641366c-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:01.069448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.069458 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70742c94-86ad-4a7f-8203-e22a7641366c-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:01.076000 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:46:01.075955 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:46:01.220767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.220725 5591 generic.go:356] "Generic (PLEG): container finished" podID="70742c94-86ad-4a7f-8203-e22a7641366c" containerID="d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae" exitCode=0 Nov 02 14:46:01.221252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.220781 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" Nov 02 14:46:01.221252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.220809 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" event={"ID":"70742c94-86ad-4a7f-8203-e22a7641366c","Type":"ContainerDied","Data":"d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae"} Nov 02 14:46:01.221252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.220855 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g" event={"ID":"70742c94-86ad-4a7f-8203-e22a7641366c","Type":"ContainerDied","Data":"a7618744c9736d538e86668f158474c04e3b3fb37dc711d359d1e35fa5fbf420"} Nov 02 14:46:01.221252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.220879 5591 scope.go:122] "RemoveContainer" containerID="d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae" Nov 02 14:46:01.232370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.232352 5591 scope.go:122] "RemoveContainer" containerID="d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae" Nov 02 14:46:01.232668 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:46:01.232648 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae\": container with ID starting with d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae not found: ID does not exist" containerID="d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae" Nov 02 14:46:01.232754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.232676 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae"} err="failed to get container status \"d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae\": rpc error: code = NotFound desc = could not find container \"d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae\": container with ID starting with d2352358d06c10dbcb50295307a168e11b35bf8c5b827989e91c6e24bf0069ae not found: ID does not exist" Nov 02 14:46:01.276040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.275999 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g"] Nov 02 14:46:01.279178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.279157 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fe1881/test-model-fe1881-kserve-785c68677c-z479g"] Nov 02 14:46:01.433629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.433589 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70742c94-86ad-4a7f-8203-e22a7641366c" path="/var/lib/kubelet/pods/70742c94-86ad-4a7f-8203-e22a7641366c/volumes" Nov 02 14:46:01.840098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:01.837616 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:03.535546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:03.532491 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:05.276967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:05.276925 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:07.017739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:07.017703 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:07.320058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:07.315755 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:07.831758 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:07.831719 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:08.675549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:08.675511 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:10.065930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:10.065888 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w"] Nov 02 14:46:10.066342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:10.066131 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" containerID="cri-o://d72b04daa3c99685f2aaad7b34bb4b2a2636f5de10adb64cd8913c64a47a2ca2" gracePeriod=30 Nov 02 14:46:10.273391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:10.273287 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:11.932585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:11.932544 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:13.581181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:13.563026 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:14.377267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:14.377233 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:14.430513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:14.430480 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl"] Nov 02 14:46:14.586132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:14.586045 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:14.746378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:14.746347 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:15.348604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.348572 5591 generic.go:356] "Generic (PLEG): container finished" podID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerID="d72b04daa3c99685f2aaad7b34bb4b2a2636f5de10adb64cd8913c64a47a2ca2" exitCode=0 Nov 02 14:46:15.348766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.348670 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" event={"ID":"69cef6fd-728f-4d68-a2d6-7818e861e12f","Type":"ContainerDied","Data":"d72b04daa3c99685f2aaad7b34bb4b2a2636f5de10adb64cd8913c64a47a2ca2"} Nov 02 14:46:15.348766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.348739 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" event={"ID":"69cef6fd-728f-4d68-a2d6-7818e861e12f","Type":"ContainerDied","Data":"52d3601f829c3616e3d5a069fd8968f99badbed3f5660e18c285af62b1cb16e3"} Nov 02 14:46:15.348766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.348759 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52d3601f829c3616e3d5a069fd8968f99badbed3f5660e18c285af62b1cb16e3" Nov 02 14:46:15.364081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.364057 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:46:15.542646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.542561 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-certs\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.542827 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.542676 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-podinfo\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.542879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.542838 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-envoy\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.542939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.542901 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-certs" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:15.543037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.542965 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-kube-api-access-6qdck\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-kube-api-access-6qdck\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.543116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543072 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-credential-socket\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.543116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543106 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69cef6fd-728f-4d68-a2d6-7818e861e12f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cef6fd-728f-4d68-a2d6-7818e861e12f-istiod-ca-cert\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.543238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543217 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-socket\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.543306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543284 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-data\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.543364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543295 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-credential-socket" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:15.543364 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543317 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-token\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-token\") pod \"69cef6fd-728f-4d68-a2d6-7818e861e12f\" (UID: \"69cef6fd-728f-4d68-a2d6-7818e861e12f\") " Nov 02 14:46:15.543643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543555 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cef6fd-728f-4d68-a2d6-7818e861e12f-istiod-ca-cert" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:46:15.543643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543582 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-socket" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:15.543895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.543672 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-data" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:15.544184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.544158 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:15.544290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.544188 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:15.544290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.544205 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:15.544290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.544219 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:15.544290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.544232 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cef6fd-728f-4d68-a2d6-7818e861e12f-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:15.545200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.545174 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-envoy" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:15.545360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.545339 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-kube-api-access-6qdck" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "kube-api-access-6qdck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:46:15.545527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.545494 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-podinfo" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:46:15.545742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.545715 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-token" pod "69cef6fd-728f-4d68-a2d6-7818e861e12f" (UID: "69cef6fd-728f-4d68-a2d6-7818e861e12f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:46:15.597853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.597801 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:15.646148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.646105 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qdck\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-kube-api-access-6qdck\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:15.646148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.646138 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:15.646148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.646147 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:15.646148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:15.646155 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cef6fd-728f-4d68-a2d6-7818e861e12f-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:16.355861 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:16.355831 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w" Nov 02 14:46:16.430310 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:16.430278 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w"] Nov 02 14:46:16.441858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:16.441814 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jw97w"] Nov 02 14:46:16.960311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:16.960278 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:17.442585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:17.442548 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" path="/var/lib/kubelet/pods/69cef6fd-728f-4d68-a2d6-7818e861e12f/volumes" Nov 02 14:46:17.678915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:17.678870 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:19.380459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:19.380426 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:19.432137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:19.432083 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" podUID="eda8bf81-a62c-460d-be49-dd42b08d7bae" containerName="payload-pre-processing" containerID="cri-o://189f3d6b3710ffaa8c5f4f370223b012d67955e61c88971874f065cf7ba14158" gracePeriod=25 Nov 02 14:46:20.219891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:20.219857 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:20.699103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:20.699067 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:20.980472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:20.980383 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:46:21.410241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:21.410196 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:21.590765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:21.590694 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:22.066253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:22.066209 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" podUID="eda8bf81-a62c-460d-be49-dd42b08d7bae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:46:22.219885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:22.219845 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:22.772923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:22.772874 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:23.539687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:23.539595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:25.247523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:25.247487 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:25.990390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:25.990352 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:26.153369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:26.153332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:26.894030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:26.893953 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:27.225227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:27.225131 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq"] Nov 02 14:46:27.225413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:27.225380 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" containerID="cri-o://05cf7483c61b086ec2f45729d6e411832966bd1b9a7ccc15d23a773ef5ea0e1d" gracePeriod=30 Nov 02 14:46:27.251348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:27.251317 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:27.418010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:27.417747 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:27.578110 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:27.578017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:28.383519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:28.383467 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:28.579520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:28.579486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:28.750169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:28.750085 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:29.045160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:29.045073 5591 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:46:29.045160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:29.045143 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:46:29.087024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:29.086966 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:29.263000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:29.262947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:30.692062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:30.692022 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:31.074860 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:46:31.074773 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:46:32.065906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.065860 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" podUID="eda8bf81-a62c-460d-be49-dd42b08d7bae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:46:32.469417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.469312 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:32.565836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.565804 5591 generic.go:356] "Generic (PLEG): container finished" podID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerID="05cf7483c61b086ec2f45729d6e411832966bd1b9a7ccc15d23a773ef5ea0e1d" exitCode=0 Nov 02 14:46:32.566023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.565882 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" event={"ID":"4d6950f9-9019-4d10-9cea-48841b9a16d5","Type":"ContainerDied","Data":"05cf7483c61b086ec2f45729d6e411832966bd1b9a7ccc15d23a773ef5ea0e1d"} Nov 02 14:46:32.566023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.565955 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" event={"ID":"4d6950f9-9019-4d10-9cea-48841b9a16d5","Type":"ContainerDied","Data":"2ff75bc611014309357beba7fc30515a9b9ff7dd1036a4859f5f55d4e388a4b7"} Nov 02 14:46:32.566134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.566020 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff75bc611014309357beba7fc30515a9b9ff7dd1036a4859f5f55d4e388a4b7" Nov 02 14:46:32.566134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.566025 5591 scope.go:122] "RemoveContainer" containerID="705fcb93444b58e530c4e16fe7c8cbde6cf2437829758b6470914fde0f6d67bb" Nov 02 14:46:32.572222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.572199 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:46:32.648225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648191 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-token\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-token\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648271 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-envoy\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648304 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-certs\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648376 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-kube-api-access-5bj7w\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-kube-api-access-5bj7w\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648446 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-socket\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648480 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-podinfo\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648511 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-data\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648664 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-certs" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:32.648736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648699 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6950f9-9019-4d10-9cea-48841b9a16d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6950f9-9019-4d10-9cea-48841b9a16d5-istiod-ca-cert\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648763 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-socket" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:32.648909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648787 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-credential-socket\") pod \"4d6950f9-9019-4d10-9cea-48841b9a16d5\" (UID: \"4d6950f9-9019-4d10-9cea-48841b9a16d5\") " Nov 02 14:46:32.648909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.648795 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-data" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:32.649142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.649007 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6950f9-9019-4d10-9cea-48841b9a16d5-istiod-ca-cert" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:46:32.649142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.649102 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-credential-socket" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:32.649588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.649556 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:32.649588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.649589 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:32.649790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.649603 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6950f9-9019-4d10-9cea-48841b9a16d5-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:32.649790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.649617 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:32.649790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.649630 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:32.650689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.650662 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-envoy" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:46:32.650874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.650852 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-kube-api-access-5bj7w" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "kube-api-access-5bj7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:46:32.650927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.650859 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-podinfo" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:46:32.650927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.650906 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-token" pod "4d6950f9-9019-4d10-9cea-48841b9a16d5" (UID: "4d6950f9-9019-4d10-9cea-48841b9a16d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:46:32.751544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.751500 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:32.751544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.751534 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bj7w\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-kube-api-access-5bj7w\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:32.751544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.751544 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:32.751784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:32.751559 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6950f9-9019-4d10-9cea-48841b9a16d5-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:33.576508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:33.576476 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq" Nov 02 14:46:33.680540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:33.680505 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq"] Nov 02 14:46:33.689593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:33.689555 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-vg6zq"] Nov 02 14:46:33.778606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:33.778515 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:33.949043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:33.948975 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:34.152004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:34.150248 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:35.435106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:35.435068 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" path="/var/lib/kubelet/pods/4d6950f9-9019-4d10-9cea-48841b9a16d5/volumes" Nov 02 14:46:35.802541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:35.802449 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:37.386443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:37.386403 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:39.055841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.055583 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:39.627430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.627377 5591 generic.go:356] "Generic (PLEG): container finished" podID="eda8bf81-a62c-460d-be49-dd42b08d7bae" containerID="189f3d6b3710ffaa8c5f4f370223b012d67955e61c88971874f065cf7ba14158" exitCode=0 Nov 02 14:46:39.627613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.627473 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" event={"ID":"eda8bf81-a62c-460d-be49-dd42b08d7bae","Type":"ContainerDied","Data":"189f3d6b3710ffaa8c5f4f370223b012d67955e61c88971874f065cf7ba14158"} Nov 02 14:46:39.719865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.719827 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:46:39.873423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.873387 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda8bf81-a62c-460d-be49-dd42b08d7bae-kube-api-access-qtcfc\" (UniqueName: \"kubernetes.io/projected/eda8bf81-a62c-460d-be49-dd42b08d7bae-kube-api-access-qtcfc\") pod \"eda8bf81-a62c-460d-be49-dd42b08d7bae\" (UID: \"eda8bf81-a62c-460d-be49-dd42b08d7bae\") " Nov 02 14:46:39.873612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.873499 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda8bf81-a62c-460d-be49-dd42b08d7bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda8bf81-a62c-460d-be49-dd42b08d7bae-plugins-config-volume\") pod \"eda8bf81-a62c-460d-be49-dd42b08d7bae\" (UID: \"eda8bf81-a62c-460d-be49-dd42b08d7bae\") " Nov 02 14:46:39.873888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.873861 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda8bf81-a62c-460d-be49-dd42b08d7bae-plugins-config-volume" pod "eda8bf81-a62c-460d-be49-dd42b08d7bae" (UID: "eda8bf81-a62c-460d-be49-dd42b08d7bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:46:39.874039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.873979 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda8bf81-a62c-460d-be49-dd42b08d7bae-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:39.875662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.875636 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8bf81-a62c-460d-be49-dd42b08d7bae-kube-api-access-qtcfc" pod "eda8bf81-a62c-460d-be49-dd42b08d7bae" (UID: "eda8bf81-a62c-460d-be49-dd42b08d7bae"). InnerVolumeSpecName "kube-api-access-qtcfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:46:39.975400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:39.975310 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtcfc\" (UniqueName: \"kubernetes.io/projected/eda8bf81-a62c-460d-be49-dd42b08d7bae-kube-api-access-qtcfc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:46:40.489200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:40.489155 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:40.667182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:40.667142 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" event={"ID":"eda8bf81-a62c-460d-be49-dd42b08d7bae","Type":"ContainerDied","Data":"ea4a3743945cb3eb956030a1e822e198cd6145e0e84c0f022d77d809a1f19546"} Nov 02 14:46:40.667489 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:40.667469 5591 scope.go:122] "RemoveContainer" containerID="189f3d6b3710ffaa8c5f4f370223b012d67955e61c88971874f065cf7ba14158" Nov 02 14:46:40.667944 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:40.667920 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl" Nov 02 14:46:40.737943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:40.737897 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl"] Nov 02 14:46:40.750973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:40.750887 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fe1881-598f47d9b7-tjwrl"] Nov 02 14:46:41.161324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:41.161284 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:41.433226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:41.433144 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda8bf81-a62c-460d-be49-dd42b08d7bae" path="/var/lib/kubelet/pods/eda8bf81-a62c-460d-be49-dd42b08d7bae/volumes" Nov 02 14:46:42.771008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:42.770438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:44.344694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:44.344657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:45.692499 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:45.692463 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:45.857112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:45.857077 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:46.059102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:46.059012 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:46.475536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:46.475496 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:46.922496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:46.922462 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:47.822667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:47.822629 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:47.984077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:47.984020 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:49.568520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:49.568484 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:51.499000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:51.495017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:52.176187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:52.176153 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:53.182210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:53.182161 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:54.771526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:54.771472 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:55.047057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:55.046497 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:55.441813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:55.441782 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:55.682256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:55.682219 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s"] Nov 02 14:46:55.857815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:55.857677 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:56.034941 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:56.034908 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:56.762712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:56.762657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:46:58.365348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:46:58.365307 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:00.065891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:00.065364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:00.254010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:00.248403 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:00.683042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:00.682955 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" podUID="60b64700-03df-4be9-9fe6-9c37acddee29" containerName="payload-pre-processing" containerID="cri-o://5f0ff6a3fa7af189a9b3fee3066db9905ae7a6df82ce9b08a4860f9c0d28ab99" gracePeriod=25 Nov 02 14:47:01.075746 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:47:01.075660 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:47:01.678722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:01.678690 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:01.847560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:01.847529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:02.510601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:02.510562 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:03.452009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:03.449434 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:05.168010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:05.156917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:05.829171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:05.829127 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" podUID="60b64700-03df-4be9-9fe6-9c37acddee29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:47:06.859561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:06.859522 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:08.472512 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:08.471045 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:10.126727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:10.126682 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:11.770326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:11.770283 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:13.463007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:13.458646 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:14.358116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:14.358084 5591 scope.go:122] "RemoveContainer" containerID="05cf7483c61b086ec2f45729d6e411832966bd1b9a7ccc15d23a773ef5ea0e1d" Nov 02 14:47:14.369187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:14.369163 5591 scope.go:122] "RemoveContainer" containerID="d72b04daa3c99685f2aaad7b34bb4b2a2636f5de10adb64cd8913c64a47a2ca2" Nov 02 14:47:15.055354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:15.055296 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:15.829026 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:15.828966 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" podUID="60b64700-03df-4be9-9fe6-9c37acddee29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:47:16.782123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:16.782078 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:18.467915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:18.467814 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:20.148446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:20.144955 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:20.621012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:20.620914 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp"] Nov 02 14:47:20.708052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:20.708020 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:21.007677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.007630 5591 generic.go:356] "Generic (PLEG): container finished" podID="60b64700-03df-4be9-9fe6-9c37acddee29" containerID="5f0ff6a3fa7af189a9b3fee3066db9905ae7a6df82ce9b08a4860f9c0d28ab99" exitCode=0 Nov 02 14:47:21.007845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.007715 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" event={"ID":"60b64700-03df-4be9-9fe6-9c37acddee29","Type":"ContainerDied","Data":"5f0ff6a3fa7af189a9b3fee3066db9905ae7a6df82ce9b08a4860f9c0d28ab99"} Nov 02 14:47:21.007845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.007823 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" event={"ID":"60b64700-03df-4be9-9fe6-9c37acddee29","Type":"ContainerDied","Data":"79374949838f5a32413c0291be6e9ff4d266901d253048208be8003db726f5df"} Nov 02 14:47:21.007998 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.007850 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79374949838f5a32413c0291be6e9ff4d266901d253048208be8003db726f5df" Nov 02 14:47:21.028604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.028577 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:47:21.157169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.157074 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b64700-03df-4be9-9fe6-9c37acddee29-kube-api-access-cnb6c\" (UniqueName: \"kubernetes.io/projected/60b64700-03df-4be9-9fe6-9c37acddee29-kube-api-access-cnb6c\") pod \"60b64700-03df-4be9-9fe6-9c37acddee29\" (UID: \"60b64700-03df-4be9-9fe6-9c37acddee29\") " Nov 02 14:47:21.157670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.157296 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b64700-03df-4be9-9fe6-9c37acddee29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b64700-03df-4be9-9fe6-9c37acddee29-plugins-config-volume\") pod \"60b64700-03df-4be9-9fe6-9c37acddee29\" (UID: \"60b64700-03df-4be9-9fe6-9c37acddee29\") " Nov 02 14:47:21.157670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.157623 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b64700-03df-4be9-9fe6-9c37acddee29-plugins-config-volume" pod "60b64700-03df-4be9-9fe6-9c37acddee29" (UID: "60b64700-03df-4be9-9fe6-9c37acddee29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:47:21.157975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.157957 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b64700-03df-4be9-9fe6-9c37acddee29-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:21.159235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.159213 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b64700-03df-4be9-9fe6-9c37acddee29-kube-api-access-cnb6c" pod "60b64700-03df-4be9-9fe6-9c37acddee29" (UID: "60b64700-03df-4be9-9fe6-9c37acddee29"). InnerVolumeSpecName "kube-api-access-cnb6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:47:21.226605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.226567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:21.259095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.259057 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnb6c\" (UniqueName: \"kubernetes.io/projected/60b64700-03df-4be9-9fe6-9c37acddee29-kube-api-access-cnb6c\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:21.883309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:21.881327 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:22.014081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:22.014045 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s" Nov 02 14:47:22.070523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:22.070483 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s"] Nov 02 14:47:22.083522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:22.083491 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cbde78-f99dbb48c-rz49s"] Nov 02 14:47:23.433574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:23.433482 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b64700-03df-4be9-9fe6-9c37acddee29" path="/var/lib/kubelet/pods/60b64700-03df-4be9-9fe6-9c37acddee29/volumes" Nov 02 14:47:23.533896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:23.533850 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:25.158015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:25.156316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:26.847663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:26.847625 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:27.565230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:27.565195 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:27.773897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:27.773854 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:47:28.542712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:28.542678 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:29.183425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:29.183290 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k"] Nov 02 14:47:29.194790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:29.194638 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5"] Nov 02 14:47:29.224461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:29.224426 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:29.392701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:29.392664 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:29.567570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:29.567472 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:30.300855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:30.300811 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:31.080437 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:47:31.080390 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:47:32.055768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:32.054005 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:33.639537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:33.639496 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:34.187793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.187745 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" podUID="5939cf19-5cac-4a71-8ea7-37fd8d4c1267" containerName="payload-processing" containerID="cri-o://93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56" gracePeriod=25 Nov 02 14:47:34.195405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.195363 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" podUID="1d843802-b544-4a9d-aaaa-046a6607e812" containerName="payload-pre-processing" containerID="cri-o://9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800" gracePeriod=25 Nov 02 14:47:34.511468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.511441 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:47:34.600649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.600613 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-kube-api-access-dvdlt\" (UniqueName: \"kubernetes.io/projected/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-kube-api-access-dvdlt\") pod \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\" (UID: \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\") " Nov 02 14:47:34.600649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.600655 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-plugins-config-volume\") pod \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\" (UID: \"5939cf19-5cac-4a71-8ea7-37fd8d4c1267\") " Nov 02 14:47:34.601054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.601028 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-plugins-config-volume" pod "5939cf19-5cac-4a71-8ea7-37fd8d4c1267" (UID: "5939cf19-5cac-4a71-8ea7-37fd8d4c1267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:47:34.601441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.601410 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:34.603132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.603108 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-kube-api-access-dvdlt" pod "5939cf19-5cac-4a71-8ea7-37fd8d4c1267" (UID: "5939cf19-5cac-4a71-8ea7-37fd8d4c1267"). InnerVolumeSpecName "kube-api-access-dvdlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:47:34.703617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:34.703579 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdlt\" (UniqueName: \"kubernetes.io/projected/5939cf19-5cac-4a71-8ea7-37fd8d4c1267-kube-api-access-dvdlt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:35.116845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.116807 5591 generic.go:356] "Generic (PLEG): container finished" podID="5939cf19-5cac-4a71-8ea7-37fd8d4c1267" containerID="93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56" exitCode=0 Nov 02 14:47:35.117095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.116851 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" Nov 02 14:47:35.117095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.116861 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" event={"ID":"5939cf19-5cac-4a71-8ea7-37fd8d4c1267","Type":"ContainerDied","Data":"93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56"} Nov 02 14:47:35.117095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.116946 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k" event={"ID":"5939cf19-5cac-4a71-8ea7-37fd8d4c1267","Type":"ContainerDied","Data":"33d8927879cdf97ead0140badb6f1709e612c998dade2dc18a9e66a445716fc3"} Nov 02 14:47:35.117095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.116976 5591 scope.go:122] "RemoveContainer" containerID="93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56" Nov 02 14:47:35.127939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.127919 5591 scope.go:122] "RemoveContainer" containerID="93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56" Nov 02 14:47:35.128224 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:47:35.128202 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56\": container with ID starting with 93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56 not found: ID does not exist" containerID="93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56" Nov 02 14:47:35.128292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.128231 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56"} err="failed to get container status \"93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56\": rpc error: code = NotFound desc = could not find container \"93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56\": container with ID starting with 93b46cb2b3257f576d6bf7f3ed13f4384b1e34d35c78bba8edbcca2708b2fe56 not found: ID does not exist" Nov 02 14:47:35.189658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.189621 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k"] Nov 02 14:47:35.202756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.202723 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-16c822-5dcf8b457-6wc6k"] Nov 02 14:47:35.392078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.392039 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:35.433196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.433161 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5939cf19-5cac-4a71-8ea7-37fd8d4c1267" path="/var/lib/kubelet/pods/5939cf19-5cac-4a71-8ea7-37fd8d4c1267/volumes" Nov 02 14:47:35.554217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.554184 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:35.643121 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.643032 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" podUID="d6d769f2-5c3b-4c9b-9795-df1032663fe6" containerName="main" containerID="cri-o://6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7" gracePeriod=45 Nov 02 14:47:35.944136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:35.944108 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:47:36.020432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.020396 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tls-certs\") pod \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " Nov 02 14:47:36.020664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.020530 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tmp-dir\") pod \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " Nov 02 14:47:36.020754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.020670 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-model-cache\") pod \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " Nov 02 14:47:36.020820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.020777 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-home\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-home\") pod \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " Nov 02 14:47:36.020883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.020865 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-dshm\") pod \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " Nov 02 14:47:36.020947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.020900 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d769f2-5c3b-4c9b-9795-df1032663fe6-kube-api-access-2lqhd\" (UniqueName: \"kubernetes.io/projected/d6d769f2-5c3b-4c9b-9795-df1032663fe6-kube-api-access-2lqhd\") pod \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\" (UID: \"d6d769f2-5c3b-4c9b-9795-df1032663fe6\") " Nov 02 14:47:36.021055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.021026 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tmp-dir" pod "d6d769f2-5c3b-4c9b-9795-df1032663fe6" (UID: "d6d769f2-5c3b-4c9b-9795-df1032663fe6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:36.021324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.021288 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-model-cache" pod "d6d769f2-5c3b-4c9b-9795-df1032663fe6" (UID: "d6d769f2-5c3b-4c9b-9795-df1032663fe6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:36.021650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.021622 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:36.021650 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.021652 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:36.021828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.021716 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-home" pod "d6d769f2-5c3b-4c9b-9795-df1032663fe6" (UID: "d6d769f2-5c3b-4c9b-9795-df1032663fe6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:36.025258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.025150 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d769f2-5c3b-4c9b-9795-df1032663fe6-kube-api-access-2lqhd" pod "d6d769f2-5c3b-4c9b-9795-df1032663fe6" (UID: "d6d769f2-5c3b-4c9b-9795-df1032663fe6"). InnerVolumeSpecName "kube-api-access-2lqhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:47:36.025258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.025229 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tls-certs" pod "d6d769f2-5c3b-4c9b-9795-df1032663fe6" (UID: "d6d769f2-5c3b-4c9b-9795-df1032663fe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:47:36.025494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.025469 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-dshm" pod "d6d769f2-5c3b-4c9b-9795-df1032663fe6" (UID: "d6d769f2-5c3b-4c9b-9795-df1032663fe6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:36.122720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.122682 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:36.122720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.122712 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d769f2-5c3b-4c9b-9795-df1032663fe6-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:36.122720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.122722 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lqhd\" (UniqueName: \"kubernetes.io/projected/d6d769f2-5c3b-4c9b-9795-df1032663fe6-kube-api-access-2lqhd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:36.123102 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.122731 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d769f2-5c3b-4c9b-9795-df1032663fe6-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:36.125656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.125623 5591 generic.go:356] "Generic (PLEG): container finished" podID="d6d769f2-5c3b-4c9b-9795-df1032663fe6" containerID="6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7" exitCode=0 Nov 02 14:47:36.125782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.125667 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" Nov 02 14:47:36.125782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.125713 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" event={"ID":"d6d769f2-5c3b-4c9b-9795-df1032663fe6","Type":"ContainerDied","Data":"6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7"} Nov 02 14:47:36.125782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.125756 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp" event={"ID":"d6d769f2-5c3b-4c9b-9795-df1032663fe6","Type":"ContainerDied","Data":"609cc63d8808753b184e24a38f8c697151d15d321f348bff922fc867a6135851"} Nov 02 14:47:36.125782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.125777 5591 scope.go:122] "RemoveContainer" containerID="6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7" Nov 02 14:47:36.137496 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.137474 5591 scope.go:122] "RemoveContainer" containerID="6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7" Nov 02 14:47:36.137781 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:47:36.137748 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7\": container with ID starting with 6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7 not found: ID does not exist" containerID="6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7" Nov 02 14:47:36.137890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.137784 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7"} err="failed to get container status \"6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7\": rpc error: code = NotFound desc = could not find container \"6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7\": container with ID starting with 6ca26bc7b6361c8e381976f9d88ba74e0cc358bcab332fbc6d623015304003a7 not found: ID does not exist" Nov 02 14:47:36.182142 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.182067 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp"] Nov 02 14:47:36.185414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:36.185391 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ea3e8f/test-model-ea3e8f-kserve-7c8c77cc59-pm7pp"] Nov 02 14:47:37.033009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:37.029345 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:37.433677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:37.433645 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d769f2-5c3b-4c9b-9795-df1032663fe6" path="/var/lib/kubelet/pods/d6d769f2-5c3b-4c9b-9795-df1032663fe6/volumes" Nov 02 14:47:38.621763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:38.621722 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:40.132349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:40.132311 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" podUID="1d843802-b544-4a9d-aaaa-046a6607e812" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:47:40.375161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:40.375120 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:42.038805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:42.038769 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:43.434610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:43.434574 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:44.157037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:44.155581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:45.875769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:45.875731 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:47.329963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:47.329925 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:48.032788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:48.032755 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:48.196169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:48.196130 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:49.792227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:49.792188 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:50.133037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:50.133000 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" podUID="1d843802-b544-4a9d-aaaa-046a6607e812" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:47:50.874143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:50.874102 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:51.041552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:51.041519 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:51.383185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:51.383078 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:51.749501 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:51.749096 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:52.046711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:52.046632 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt"] Nov 02 14:47:52.047177 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:52.046843 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" containerID="cri-o://9c5c3f770e1a620b17a353763bd22042fdc0ee13f59115f3dade7659ac7ede7f" gracePeriod=30 Nov 02 14:47:52.132765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:52.132714 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:53.028737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:53.028706 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:53.189647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:53.189614 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:53.348269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:53.348184 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:53.816417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:53.816373 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:54.117495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.117405 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42"] Nov 02 14:47:54.117772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.117734 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" containerID="cri-o://56416d2f8eaa7f5768a647c01f49e347ec40083d42f5f351267526c541087092" gracePeriod=30 Nov 02 14:47:54.248509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.248474 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:54.522061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.522030 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:54.536448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.536419 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:47:54.651088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.651053 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d843802-b544-4a9d-aaaa-046a6607e812-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d843802-b544-4a9d-aaaa-046a6607e812-plugins-config-volume\") pod \"1d843802-b544-4a9d-aaaa-046a6607e812\" (UID: \"1d843802-b544-4a9d-aaaa-046a6607e812\") " Nov 02 14:47:54.651289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.651169 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d843802-b544-4a9d-aaaa-046a6607e812-kube-api-access-9zj2k\" (UniqueName: \"kubernetes.io/projected/1d843802-b544-4a9d-aaaa-046a6607e812-kube-api-access-9zj2k\") pod \"1d843802-b544-4a9d-aaaa-046a6607e812\" (UID: \"1d843802-b544-4a9d-aaaa-046a6607e812\") " Nov 02 14:47:54.651467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.651444 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d843802-b544-4a9d-aaaa-046a6607e812-plugins-config-volume" pod "1d843802-b544-4a9d-aaaa-046a6607e812" (UID: "1d843802-b544-4a9d-aaaa-046a6607e812"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:47:54.651628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.651614 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d843802-b544-4a9d-aaaa-046a6607e812-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:54.653296 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.653273 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d843802-b544-4a9d-aaaa-046a6607e812-kube-api-access-9zj2k" pod "1d843802-b544-4a9d-aaaa-046a6607e812" (UID: "1d843802-b544-4a9d-aaaa-046a6607e812"). InnerVolumeSpecName "kube-api-access-9zj2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:47:54.753314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:54.753274 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zj2k\" (UniqueName: \"kubernetes.io/projected/1d843802-b544-4a9d-aaaa-046a6607e812-kube-api-access-9zj2k\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:55.180312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.180274 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:55.346803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.346765 5591 generic.go:356] "Generic (PLEG): container finished" podID="1d843802-b544-4a9d-aaaa-046a6607e812" containerID="9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800" exitCode=0 Nov 02 14:47:55.346803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.346805 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" Nov 02 14:47:55.347370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.346842 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" event={"ID":"1d843802-b544-4a9d-aaaa-046a6607e812","Type":"ContainerDied","Data":"9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800"} Nov 02 14:47:55.347370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.346898 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5" event={"ID":"1d843802-b544-4a9d-aaaa-046a6607e812","Type":"ContainerDied","Data":"669730f1f6e755865e2e5481191716c7604d4920d2d567a5b3af067fc13f7372"} Nov 02 14:47:55.347370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.346924 5591 scope.go:122] "RemoveContainer" containerID="9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800" Nov 02 14:47:55.358734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.358708 5591 scope.go:122] "RemoveContainer" containerID="9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800" Nov 02 14:47:55.359101 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:47:55.359073 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800\": container with ID starting with 9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800 not found: ID does not exist" containerID="9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800" Nov 02 14:47:55.359175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.359110 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800"} err="failed to get container status \"9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800\": rpc error: code = NotFound desc = could not find container \"9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800\": container with ID starting with 9072921c4848c719a960707bf4cbed06594183dc5e8caa1ef031be6a38c55800 not found: ID does not exist" Nov 02 14:47:55.364811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.364789 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:55.420963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.420931 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5"] Nov 02 14:47:55.439246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439158 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg"] Nov 02 14:47:55.439888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439864 5591 cpu_manager.go:414] "containerMap: removing container" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:47:55.439888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439883 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439896 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439902 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439909 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439914 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439921 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439925 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439932 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1d843802-b544-4a9d-aaaa-046a6607e812" containerName="payload-pre-processing" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439937 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d843802-b544-4a9d-aaaa-046a6607e812" containerName="payload-pre-processing" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439947 5591 cpu_manager.go:414] "containerMap: removing container" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439953 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439965 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439969 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439977 5591 cpu_manager.go:414] "containerMap: removing container" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.439996 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440003 5591 cpu_manager.go:414] "containerMap: removing container" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440008 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440014 5591 cpu_manager.go:414] "containerMap: removing container" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440018 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440026 5591 cpu_manager.go:414] "containerMap: removing container" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440030 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440038 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440043 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440052 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" containerName="payload-pre-processing" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440057 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" containerName="payload-pre-processing" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440065 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440069 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440075 5591 cpu_manager.go:414] "containerMap: removing container" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440079 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440085 5591 cpu_manager.go:414] "containerMap: removing container" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440090 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440098 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440103 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440108 5591 cpu_manager.go:414] "containerMap: removing container" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440114 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440124 5591 cpu_manager.go:414] "containerMap: removing container" podUID="70742c94-86ad-4a7f-8203-e22a7641366c" containerName="main" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440128 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70742c94-86ad-4a7f-8203-e22a7641366c" containerName="main" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440136 5591 cpu_manager.go:414] "containerMap: removing container" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" Nov 02 14:47:55.440114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440141 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440150 5591 cpu_manager.go:414] "containerMap: removing container" podUID="60b64700-03df-4be9-9fe6-9c37acddee29" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440154 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b64700-03df-4be9-9fe6-9c37acddee29" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440164 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440168 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440176 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d6d769f2-5c3b-4c9b-9795-df1032663fe6" containerName="main" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440180 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d769f2-5c3b-4c9b-9795-df1032663fe6" containerName="main" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440188 5591 cpu_manager.go:414] "containerMap: removing container" podUID="54e490c6-5154-4333-be08-3bf46465f4de" containerName="payload-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440192 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e490c6-5154-4333-be08-3bf46465f4de" containerName="payload-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440201 5591 cpu_manager.go:414] "containerMap: removing container" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440206 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440213 5591 cpu_manager.go:414] "containerMap: removing container" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440218 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440226 5591 cpu_manager.go:414] "containerMap: removing container" podUID="eda8bf81-a62c-460d-be49-dd42b08d7bae" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440230 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda8bf81-a62c-460d-be49-dd42b08d7bae" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440240 5591 cpu_manager.go:414] "containerMap: removing container" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440245 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440253 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440257 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440265 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440269 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440277 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5939cf19-5cac-4a71-8ea7-37fd8d4c1267" containerName="payload-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440282 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5939cf19-5cac-4a71-8ea7-37fd8d4c1267" containerName="payload-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440289 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440294 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440304 5591 cpu_manager.go:414] "containerMap: removing container" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440308 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440316 5591 cpu_manager.go:414] "containerMap: removing container" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440320 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440326 5591 cpu_manager.go:414] "containerMap: removing container" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440330 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440339 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440344 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440355 5591 cpu_manager.go:414] "containerMap: removing container" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440359 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440367 5591 cpu_manager.go:414] "containerMap: removing container" podUID="64bb2eb4-559d-4b7c-8833-a76ba98da86e" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440372 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bb2eb4-559d-4b7c-8833-a76ba98da86e" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440380 5591 cpu_manager.go:414] "containerMap: removing container" podUID="825bfd14-a3a6-4436-9286-3f3ea1f3fd49" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440385 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825bfd14-a3a6-4436-9286-3f3ea1f3fd49" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440391 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440395 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440402 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440406 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440413 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440417 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440425 5591 cpu_manager.go:414] "containerMap: removing container" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440430 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440437 5591 cpu_manager.go:414] "containerMap: removing container" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440441 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440545 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440554 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440560 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4529748-2a47-4b75-b885-0a83f37a0705" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440566 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440572 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440577 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5939cf19-5cac-4a71-8ea7-37fd8d4c1267" containerName="payload-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440585 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440591 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440596 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440602 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440606 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440612 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2fb505-7f16-449b-9b80-ae516cf2bba7" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440619 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440624 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440629 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bb2eb4-559d-4b7c-8833-a76ba98da86e" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440635 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440641 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="825bfd14-a3a6-4436-9286-3f3ea1f3fd49" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440647 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440652 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440657 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440664 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440670 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440676 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440682 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440688 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440692 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440698 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440702 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440707 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d843802-b544-4a9d-aaaa-046a6607e812" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440713 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440719 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440724 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440729 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440736 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440741 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebeafe1a-8227-4ab6-9cd7-9f5f3ba75d2c" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440747 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440752 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440757 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f48a2c-7440-4363-bfc0-a7ccafdbed80" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440764 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440770 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="70742c94-86ad-4a7f-8203-e22a7641366c" containerName="main" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440775 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="69cef6fd-728f-4d68-a2d6-7818e861e12f" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440781 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b64700-03df-4be9-9fe6-9c37acddee29" containerName="payload-pre-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440787 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440793 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d769f2-5c3b-4c9b-9795-df1032663fe6" containerName="main" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440798 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e490c6-5154-4333-be08-3bf46465f4de" containerName="payload-processing" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440802 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440809 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" Nov 02 14:47:55.442221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.440815 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda8bf81-a62c-460d-be49-dd42b08d7bae" containerName="payload-pre-processing" Nov 02 14:47:55.447715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.446093 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.447715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.445898 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-16c822-7b9889858c-2hqz5"] Nov 02 14:47:55.452642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.452616 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg"] Nov 02 14:47:55.564301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564261 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11918adc-cce0-49a6-9f2b-30ea03866457-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.564301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564302 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7fkp\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-kube-api-access-f7fkp\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.564547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564332 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.564547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564436 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.564620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564557 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11918adc-cce0-49a6-9f2b-30ea03866457-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.564663 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564620 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.564713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564680 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.564752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564727 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.564798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.564761 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667042 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11918adc-cce0-49a6-9f2b-30ea03866457-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667111 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667142 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667184 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667225 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667314 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11918adc-cce0-49a6-9f2b-30ea03866457-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667347 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7fkp\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-kube-api-access-f7fkp\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667383 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667614 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667624 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667620 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667850 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667840 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.667996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.667968 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11918adc-cce0-49a6-9f2b-30ea03866457-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.668220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.668198 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.669717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.669690 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11918adc-cce0-49a6-9f2b-30ea03866457-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.669839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.669816 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.692400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.692334 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.692521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.692405 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7fkp\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-kube-api-access-f7fkp\") pod \"maas-default-gateway-openshift-default-6886b96f9-4rwmg\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:55.764926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:55.764869 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:56.096029 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:47:56.095949 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11918adc_cce0_49a6_9f2b_30ea03866457.slice/crio-6343ee9e5e94e55cfd52378bd43d777d955251f57628cca7987d199a79522f74 WatchSource:0}: Error finding container 6343ee9e5e94e55cfd52378bd43d777d955251f57628cca7987d199a79522f74: Status 404 returned error can't find the container with id 6343ee9e5e94e55cfd52378bd43d777d955251f57628cca7987d199a79522f74 Nov 02 14:47:56.096432 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.096408 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg"] Nov 02 14:47:56.099071 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.099033 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:47:56.099195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.099113 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:47:56.099195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.099182 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:47:56.319748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.319700 5591 patch_prober.go:29] interesting pod/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:47:56.319926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.319793 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:47:56.364919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.364818 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" event={"ID":"11918adc-cce0-49a6-9f2b-30ea03866457","Type":"ContainerStarted","Data":"8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc"} Nov 02 14:47:56.364919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.364892 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" event={"ID":"11918adc-cce0-49a6-9f2b-30ea03866457","Type":"ContainerStarted","Data":"6343ee9e5e94e55cfd52378bd43d777d955251f57628cca7987d199a79522f74"} Nov 02 14:47:56.423972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.423901 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podStartSLOduration=1.4238832289999999 podStartE2EDuration="1.423883229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:47:56.421468534 +0000 UTC m=+3825.147090302" watchObservedRunningTime="2026-11-02 14:47:56.423883229 +0000 UTC m=+3825.149505000" Nov 02 14:47:56.765421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:56.765386 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:47:57.175081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.174480 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:57.373545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.373512 5591 generic.go:356] "Generic (PLEG): container finished" podID="606cd572-014b-499d-8622-67bfad54d0e0" containerID="9c5c3f770e1a620b17a353763bd22042fdc0ee13f59115f3dade7659ac7ede7f" exitCode=0 Nov 02 14:47:57.374046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.373586 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" event={"ID":"606cd572-014b-499d-8622-67bfad54d0e0","Type":"ContainerDied","Data":"9c5c3f770e1a620b17a353763bd22042fdc0ee13f59115f3dade7659ac7ede7f"} Nov 02 14:47:57.374046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.373645 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" event={"ID":"606cd572-014b-499d-8622-67bfad54d0e0","Type":"ContainerDied","Data":"05d23814fb16000196f2b00c120869c670cf1e25bfeccfdc578df4d0c8603712"} Nov 02 14:47:57.374046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.373668 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d23814fb16000196f2b00c120869c670cf1e25bfeccfdc578df4d0c8603712" Nov 02 14:47:57.374046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.373687 5591 scope.go:122] "RemoveContainer" containerID="7c864929490bf6c1abec5e8566fcdff7b5e2f3f0820593cbc4cdcbdbc692b2a1" Nov 02 14:47:57.397306 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:47:57.397280 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c864929490bf6c1abec5e8566fcdff7b5e2f3f0820593cbc4cdcbdbc692b2a1\": container with ID starting with 7c864929490bf6c1abec5e8566fcdff7b5e2f3f0820593cbc4cdcbdbc692b2a1 not found: ID does not exist" containerID="7c864929490bf6c1abec5e8566fcdff7b5e2f3f0820593cbc4cdcbdbc692b2a1" Nov 02 14:47:57.400943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.400921 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:47:57.431967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.431933 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d843802-b544-4a9d-aaaa-046a6607e812" path="/var/lib/kubelet/pods/1d843802-b544-4a9d-aaaa-046a6607e812/volumes" Nov 02 14:47:57.496847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.496816 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606cd572-014b-499d-8622-67bfad54d0e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606cd572-014b-499d-8622-67bfad54d0e0-istiod-ca-cert\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.497069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.496860 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-envoy\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.497069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.496891 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/606cd572-014b-499d-8622-67bfad54d0e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606cd572-014b-499d-8622-67bfad54d0e0-istio-podinfo\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.497223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497071 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-kube-api-access-xn5gc\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-kube-api-access-xn5gc\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.497297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497268 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606cd572-014b-499d-8622-67bfad54d0e0-istiod-ca-cert" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:47:57.497361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497280 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-istio-token\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-istio-token\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.497430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497390 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-data\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.497507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497489 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-certs\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.497699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497667 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-data" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:57.497841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497822 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-certs" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:57.497900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497871 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-credential-socket\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.497900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.497895 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-socket\") pod \"606cd572-014b-499d-8622-67bfad54d0e0\" (UID: \"606cd572-014b-499d-8622-67bfad54d0e0\") " Nov 02 14:47:57.498175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.498145 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-credential-socket" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:57.498299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.498250 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-socket" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:57.498695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.498673 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.498799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.498698 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.498799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.498712 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.498799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.498725 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606cd572-014b-499d-8622-67bfad54d0e0-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.498799 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.498738 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.499818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.499786 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-istio-token" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:47:57.499961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.499938 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/606cd572-014b-499d-8622-67bfad54d0e0-istio-podinfo" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:47:57.500959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.500932 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-kube-api-access-xn5gc" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "kube-api-access-xn5gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:47:57.501113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.501083 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-envoy" pod "606cd572-014b-499d-8622-67bfad54d0e0" (UID: "606cd572-014b-499d-8622-67bfad54d0e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:57.600260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.600215 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.600260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.600258 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606cd572-014b-499d-8622-67bfad54d0e0-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.600260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.600268 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606cd572-014b-499d-8622-67bfad54d0e0-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.600505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.600278 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5gc\" (UniqueName: \"kubernetes.io/projected/606cd572-014b-499d-8622-67bfad54d0e0-kube-api-access-xn5gc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:57.766618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.766576 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4rwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 02 14:47:57.766818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:57.766635 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 02 14:47:58.383282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:58.383248 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt" Nov 02 14:47:58.465668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:58.465569 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt"] Nov 02 14:47:58.475036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:58.474996 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-16c822-openshift-default-84f559df4c-lr5mt"] Nov 02 14:47:58.766018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:58.765901 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4rwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 02 14:47:58.766018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:58.765953 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 02 14:47:58.970009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:58.969139 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:47:59.396425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.396338 5591 generic.go:356] "Generic (PLEG): container finished" podID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerID="56416d2f8eaa7f5768a647c01f49e347ec40083d42f5f351267526c541087092" exitCode=0 Nov 02 14:47:59.396425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.396395 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" event={"ID":"bd282e44-172b-4b31-81ce-ddfc8fd63a78","Type":"ContainerDied","Data":"56416d2f8eaa7f5768a647c01f49e347ec40083d42f5f351267526c541087092"} Nov 02 14:47:59.396905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.396486 5591 scope.go:122] "RemoveContainer" containerID="6b8fc3c7e0e74e5ee876768bb8822f005f6c476adab14f612ed9292e07cc8a7d" Nov 02 14:47:59.472679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.472639 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606cd572-014b-499d-8622-67bfad54d0e0" path="/var/lib/kubelet/pods/606cd572-014b-499d-8622-67bfad54d0e0/volumes" Nov 02 14:47:59.480242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.480217 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:47:59.527216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527166 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-kube-api-access-2dpx4\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-kube-api-access-2dpx4\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.527431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527265 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-credential-socket\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.527431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527360 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-envoy\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.527580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527461 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-certs\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.527580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527570 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-socket\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.527753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527612 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-token\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-token\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.527753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527684 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-data\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.527753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527732 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istiod-ca-cert\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.528384 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.527781 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-podinfo\") pod \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\" (UID: \"bd282e44-172b-4b31-81ce-ddfc8fd63a78\") " Nov 02 14:47:59.530162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.529703 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-data" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:59.530162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.529901 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-credential-socket" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:59.530162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.529977 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istiod-ca-cert" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:47:59.530162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.530014 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-certs" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:59.530162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.530071 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-socket" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:59.562266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.562178 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-token" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:47:59.564035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.563677 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-podinfo" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:47:59.567405 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.567365 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-envoy" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:47:59.570746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.570715 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-kube-api-access-2dpx4" pod "bd282e44-172b-4b31-81ce-ddfc8fd63a78" (UID: "bd282e44-172b-4b31-81ce-ddfc8fd63a78"). InnerVolumeSpecName "kube-api-access-2dpx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:47:59.629553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629512 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.629553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629547 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.629553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629556 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.629814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629565 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.629814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629587 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.629814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629596 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dpx4\" (UniqueName: \"kubernetes.io/projected/bd282e44-172b-4b31-81ce-ddfc8fd63a78-kube-api-access-2dpx4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.629814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629606 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.629814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629615 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.629814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.629624 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd282e44-172b-4b31-81ce-ddfc8fd63a78-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:47:59.765959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.765871 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4rwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 02 14:47:59.765959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.765928 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 02 14:47:59.920061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.920026 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5"] Nov 02 14:47:59.960570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:47:59.960535 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4"] Nov 02 14:48:00.108387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.108302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:00.280375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.280335 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:00.407667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.407637 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" Nov 02 14:48:00.408218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.407681 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42" event={"ID":"bd282e44-172b-4b31-81ce-ddfc8fd63a78","Type":"ContainerDied","Data":"fe9e5285fc3fbaa4b704507e1aea0cdfb15e391968be3fcc2fcf147dae1fd0a6"} Nov 02 14:48:00.408218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.407753 5591 scope.go:122] "RemoveContainer" containerID="56416d2f8eaa7f5768a647c01f49e347ec40083d42f5f351267526c541087092" Nov 02 14:48:00.457502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.457471 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:00.480658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.480626 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42"] Nov 02 14:48:00.494271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.494234 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cbde78-openshift-default-75649f999f-2rs42"] Nov 02 14:48:00.765805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.765722 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4rwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 02 14:48:00.765805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:00.765790 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 02 14:48:01.106195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:01.106054 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:01.108208 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:01.107174 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:48:01.463628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:01.463592 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" path="/var/lib/kubelet/pods/bd282e44-172b-4b31-81ce-ddfc8fd63a78/volumes" Nov 02 14:48:01.765949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:01.765858 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4rwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 02 14:48:01.765949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:01.765915 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 02 14:48:02.765871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:02.765822 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4rwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 02 14:48:02.766328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:02.765898 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 02 14:48:02.871338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:02.871298 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:03.765441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:03.765400 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4rwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 02 14:48:03.765623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:03.765473 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 02 14:48:04.446640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:04.446593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:04.921493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:04.921440 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" podUID="07dac19e-1503-410b-8eab-6905bb48ec0b" containerName="payload-processing" containerID="cri-o://b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107" gracePeriod=25 Nov 02 14:48:04.961833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:04.961790 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" podUID="ee57b283-ffc3-4310-9113-6568a46d0034" containerName="payload-pre-processing" containerID="cri-o://3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39" gracePeriod=25 Nov 02 14:48:05.215319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:05.215221 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:48:05.215856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:05.215833 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:48:05.217182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:05.217156 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:48:05.601113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:05.601025 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89"] Nov 02 14:48:05.726597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:05.726560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:06.266805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:06.266767 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:07.014202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:07.014168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:07.878032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:07.877969 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" podUID="07dac19e-1503-410b-8eab-6905bb48ec0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:48:08.005821 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:08.005782 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:08.886727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:08.886681 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" podUID="ee57b283-ffc3-4310-9113-6568a46d0034" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:48:09.657011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:09.656236 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:10.454005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.453956 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g"] Nov 02 14:48:10.454698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454681 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:48:10.454760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454699 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:48:10.454760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454715 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" Nov 02 14:48:10.454760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454720 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d71f880-d4ea-4d6d-9505-bccb5c82253b" containerName="istio-proxy" Nov 02 14:48:10.454760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454734 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" Nov 02 14:48:10.454760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454739 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" Nov 02 14:48:10.454760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454749 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" Nov 02 14:48:10.454760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454753 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" Nov 02 14:48:10.454760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454761 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454766 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0398bb65-9c17-4f32-b0cf-e2f0c8e56dd8" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454784 5591 cpu_manager.go:414] "containerMap: removing container" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454788 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454796 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454801 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59d93eb-2fa4-4094-98df-b0f1d75e0c94" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454820 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454825 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454835 5591 cpu_manager.go:414] "containerMap: removing container" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454839 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454846 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454851 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6950f9-9019-4d10-9cea-48841b9a16d5" containerName="istio-proxy" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454858 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454862 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454869 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454874 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454881 5591 cpu_manager.go:414] "containerMap: removing container" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454885 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd174ecd-1998-4f54-8588-9de900552feb" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454893 5591 cpu_manager.go:414] "containerMap: removing container" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454898 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454906 5591 cpu_manager.go:414] "containerMap: removing container" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454911 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454917 5591 cpu_manager.go:414] "containerMap: removing container" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.454921 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91775710-4a1d-4dbb-ab4d-d3c771f4cad2" containerName="cleanup" Nov 02 14:48:10.455036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455044 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7a7fb7-de79-44d6-ac70-753b949d5a12" containerName="cleanup" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455057 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="62221ac5-75fe-4999-9df5-2952cd3dee73" containerName="cleanup" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455066 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6982912-1475-49a1-a605-2db8b5bc7687" containerName="cleanup" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455074 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0947e44-e37f-4591-adfe-714233cdbd98" containerName="cleanup" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455079 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f73d79-17ec-4ba0-a85c-08ba05fd9af6" containerName="cleanup" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455087 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="96800fad-b801-4f01-91db-db678d597bf1" containerName="cleanup" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455093 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d03007-f6a2-47d4-8742-4ca9d4d2f71c" containerName="cleanup" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455100 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ce129d-05f8-48db-82cb-fa836c2992a0" containerName="cleanup" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455105 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455115 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455120 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd282e44-172b-4b31-81ce-ddfc8fd63a78" containerName="istio-proxy" Nov 02 14:48:10.455803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.455129 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="606cd572-014b-499d-8622-67bfad54d0e0" containerName="istio-proxy" Nov 02 14:48:10.461406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.461374 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.486645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.486612 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g"] Nov 02 14:48:10.540331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540291 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.540518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540337 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.540518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540457 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wplc\" (UniqueName: \"kubernetes.io/projected/cf93ec7a-9a28-4e39-b0e2-79e786e431de-kube-api-access-9wplc\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.540620 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540597 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.540713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540696 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.540774 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540760 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.540842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540822 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.540910 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540896 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.540957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.540946 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642350 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642388 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642387 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642436 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wplc\" (UniqueName: \"kubernetes.io/projected/cf93ec7a-9a28-4e39-b0e2-79e786e431de-kube-api-access-9wplc\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642519 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642598 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642696 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642722 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642868 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642893 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.642961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.642933 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.643170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.643105 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.643170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.643127 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.643279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.643203 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.643450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.643432 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.644839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.644814 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.645406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.645382 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.667343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.667308 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf93ec7a-9a28-4e39-b0e2-79e786e431de-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.667557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.667532 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wplc\" (UniqueName: \"kubernetes.io/projected/cf93ec7a-9a28-4e39-b0e2-79e786e431de-kube-api-access-9wplc\") pod \"maas-default-gateway-openshift-default-6886b96f9-xqg2g\" (UID: \"cf93ec7a-9a28-4e39-b0e2-79e786e431de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:10.787867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:10.787779 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:11.126172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.126136 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g"] Nov 02 14:48:11.126603 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:48:11.126580 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf93ec7a_9a28_4e39_b0e2_79e786e431de.slice/crio-14bc15474ab6529b9a091dbd537b015b4ec3e0586d59e1c382941538f13dadd4 WatchSource:0}: Error finding container 14bc15474ab6529b9a091dbd537b015b4ec3e0586d59e1c382941538f13dadd4: Status 404 returned error can't find the container with id 14bc15474ab6529b9a091dbd537b015b4ec3e0586d59e1c382941538f13dadd4 Nov 02 14:48:11.129602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.129562 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:48:11.129713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.129640 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:48:11.129713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.129688 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:48:11.406080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.395724 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:11.515596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.515548 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" event={"ID":"cf93ec7a-9a28-4e39-b0e2-79e786e431de","Type":"ContainerStarted","Data":"ea851acb5cad3e61892cf0f937495662c935f12a532a24a7cc5c8d2d3a67333e"} Nov 02 14:48:11.516170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.515611 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" event={"ID":"cf93ec7a-9a28-4e39-b0e2-79e786e431de","Type":"ContainerStarted","Data":"14bc15474ab6529b9a091dbd537b015b4ec3e0586d59e1c382941538f13dadd4"} Nov 02 14:48:11.554728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.554691 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:11.598570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.598510 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podStartSLOduration=1.598495238 podStartE2EDuration="1.598495238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:11.58369184 +0000 UTC m=+3840.309313609" watchObservedRunningTime="2026-11-02 14:48:11.598495238 +0000 UTC m=+3840.324117008" Nov 02 14:48:11.788020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:11.787914 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:12.789923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:12.789882 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 02 14:48:12.790438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:12.789939 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 02 14:48:13.073586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:13.071115 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:13.251008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:13.245202 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:13.788383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:13.788349 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 02 14:48:13.788586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:13.788403 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 02 14:48:13.987195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:13.987160 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:14.158282 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:14.158238 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:14.412789 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:14.412710 5591 scope.go:122] "RemoveContainer" containerID="9c5c3f770e1a620b17a353763bd22042fdc0ee13f59115f3dade7659ac7ede7f" Nov 02 14:48:14.424029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:14.423999 5591 scope.go:122] "RemoveContainer" containerID="5f0ff6a3fa7af189a9b3fee3066db9905ae7a6df82ce9b08a4860f9c0d28ab99" Nov 02 14:48:14.799207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:14.799094 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 02 14:48:14.799207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:14.799166 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 02 14:48:14.808769 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:14.808725 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:15.788251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:15.788211 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 02 14:48:15.788665 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:15.788268 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 02 14:48:16.573010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:16.564519 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:16.788970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:16.788924 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 02 14:48:16.789434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:16.789014 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 02 14:48:17.788352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:17.788312 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 02 14:48:17.788552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:17.788381 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 02 14:48:17.878478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:17.878441 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" podUID="07dac19e-1503-410b-8eab-6905bb48ec0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:48:17.882862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:17.882831 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:18.275804 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:18.275760 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:18.788527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:18.788491 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 02 14:48:18.788721 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:18.788560 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 02 14:48:18.886008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:18.885952 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" podUID="ee57b283-ffc3-4310-9113-6568a46d0034" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:48:19.793844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:19.793812 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:19.794312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:19.794298 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:19.795079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:19.795054 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" Nov 02 14:48:19.839584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:19.839546 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:20.624368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:20.624323 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" podUID="c782b989-1334-4b2d-a7dd-1080b0951ea7" containerName="main" containerID="cri-o://d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6" gracePeriod=45 Nov 02 14:48:20.903266 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:20.903239 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:48:21.019806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.019771 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-dshm\") pod \"c782b989-1334-4b2d-a7dd-1080b0951ea7\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " Nov 02 14:48:21.020034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.019865 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-home\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-home\") pod \"c782b989-1334-4b2d-a7dd-1080b0951ea7\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " Nov 02 14:48:21.020034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.019938 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c782b989-1334-4b2d-a7dd-1080b0951ea7-kube-api-access-wdpp8\" (UniqueName: \"kubernetes.io/projected/c782b989-1334-4b2d-a7dd-1080b0951ea7-kube-api-access-wdpp8\") pod \"c782b989-1334-4b2d-a7dd-1080b0951ea7\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " Nov 02 14:48:21.020185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020049 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-tmp-dir\") pod \"c782b989-1334-4b2d-a7dd-1080b0951ea7\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " Nov 02 14:48:21.020185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020089 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-model-cache\") pod \"c782b989-1334-4b2d-a7dd-1080b0951ea7\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " Nov 02 14:48:21.020185 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020123 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c782b989-1334-4b2d-a7dd-1080b0951ea7-tls-certs\" (UniqueName: \"kubernetes.io/secret/c782b989-1334-4b2d-a7dd-1080b0951ea7-tls-certs\") pod \"c782b989-1334-4b2d-a7dd-1080b0951ea7\" (UID: \"c782b989-1334-4b2d-a7dd-1080b0951ea7\") " Nov 02 14:48:21.020360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020196 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-home" pod "c782b989-1334-4b2d-a7dd-1080b0951ea7" (UID: "c782b989-1334-4b2d-a7dd-1080b0951ea7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:21.020360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020314 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-tmp-dir" pod "c782b989-1334-4b2d-a7dd-1080b0951ea7" (UID: "c782b989-1334-4b2d-a7dd-1080b0951ea7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:21.020483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020398 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-model-cache" pod "c782b989-1334-4b2d-a7dd-1080b0951ea7" (UID: "c782b989-1334-4b2d-a7dd-1080b0951ea7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:21.020828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020804 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:21.020940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020830 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:21.020940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.020840 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:21.022341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.022315 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-dshm" pod "c782b989-1334-4b2d-a7dd-1080b0951ea7" (UID: "c782b989-1334-4b2d-a7dd-1080b0951ea7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:21.022458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.022445 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c782b989-1334-4b2d-a7dd-1080b0951ea7-kube-api-access-wdpp8" pod "c782b989-1334-4b2d-a7dd-1080b0951ea7" (UID: "c782b989-1334-4b2d-a7dd-1080b0951ea7"). InnerVolumeSpecName "kube-api-access-wdpp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:48:21.022626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.022611 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c782b989-1334-4b2d-a7dd-1080b0951ea7-tls-certs" pod "c782b989-1334-4b2d-a7dd-1080b0951ea7" (UID: "c782b989-1334-4b2d-a7dd-1080b0951ea7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:48:21.121997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.121956 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c782b989-1334-4b2d-a7dd-1080b0951ea7-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:21.121997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.121998 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c782b989-1334-4b2d-a7dd-1080b0951ea7-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:21.122200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.122010 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdpp8\" (UniqueName: \"kubernetes.io/projected/c782b989-1334-4b2d-a7dd-1080b0951ea7-kube-api-access-wdpp8\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:21.619541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.619506 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:21.646260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.646227 5591 generic.go:356] "Generic (PLEG): container finished" podID="c782b989-1334-4b2d-a7dd-1080b0951ea7" containerID="d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6" exitCode=0 Nov 02 14:48:21.646754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.646271 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" Nov 02 14:48:21.646754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.646311 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" event={"ID":"c782b989-1334-4b2d-a7dd-1080b0951ea7","Type":"ContainerDied","Data":"d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6"} Nov 02 14:48:21.646754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.646354 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89" event={"ID":"c782b989-1334-4b2d-a7dd-1080b0951ea7","Type":"ContainerDied","Data":"4c3a48471e41fed92d2afaf838e8f520e97f9b5ba07e3c11b67b176d56d6ae37"} Nov 02 14:48:21.646754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.646375 5591 scope.go:122] "RemoveContainer" containerID="d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6" Nov 02 14:48:21.656844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.656818 5591 scope.go:122] "RemoveContainer" containerID="d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6" Nov 02 14:48:21.657207 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:21.657185 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6\": container with ID starting with d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6 not found: ID does not exist" containerID="d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6" Nov 02 14:48:21.657290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.657215 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6"} err="failed to get container status \"d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6\": rpc error: code = NotFound desc = could not find container \"d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6\": container with ID starting with d4d9e149d072d21e87a1e6a88668980a2be1d266aac7561e70f8c90d80307bc6 not found: ID does not exist" Nov 02 14:48:21.702171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.702076 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89"] Nov 02 14:48:21.705577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:21.705553 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c8f647/test-model-c8f647-kserve-7d874f787f-7pz89"] Nov 02 14:48:23.240301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:23.239732 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:23.432977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:23.432939 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c782b989-1334-4b2d-a7dd-1080b0951ea7" path="/var/lib/kubelet/pods/c782b989-1334-4b2d-a7dd-1080b0951ea7/volumes" Nov 02 14:48:24.818925 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:24.818884 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:25.070245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.070162 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf"] Nov 02 14:48:25.249859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.244356 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:25.360443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.360403 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:48:25.439566 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.439527 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:48:25.497559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.497522 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07dac19e-1503-410b-8eab-6905bb48ec0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dac19e-1503-410b-8eab-6905bb48ec0b-plugins-config-volume\") pod \"07dac19e-1503-410b-8eab-6905bb48ec0b\" (UID: \"07dac19e-1503-410b-8eab-6905bb48ec0b\") " Nov 02 14:48:25.497828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.497608 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee57b283-ffc3-4310-9113-6568a46d0034-kube-api-access-2rww9\" (UniqueName: \"kubernetes.io/projected/ee57b283-ffc3-4310-9113-6568a46d0034-kube-api-access-2rww9\") pod \"ee57b283-ffc3-4310-9113-6568a46d0034\" (UID: \"ee57b283-ffc3-4310-9113-6568a46d0034\") " Nov 02 14:48:25.497828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.497646 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee57b283-ffc3-4310-9113-6568a46d0034-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee57b283-ffc3-4310-9113-6568a46d0034-plugins-config-volume\") pod \"ee57b283-ffc3-4310-9113-6568a46d0034\" (UID: \"ee57b283-ffc3-4310-9113-6568a46d0034\") " Nov 02 14:48:25.497828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.497791 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07dac19e-1503-410b-8eab-6905bb48ec0b-kube-api-access-4s6ct\" (UniqueName: \"kubernetes.io/projected/07dac19e-1503-410b-8eab-6905bb48ec0b-kube-api-access-4s6ct\") pod \"07dac19e-1503-410b-8eab-6905bb48ec0b\" (UID: \"07dac19e-1503-410b-8eab-6905bb48ec0b\") " Nov 02 14:48:25.498557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.498254 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee57b283-ffc3-4310-9113-6568a46d0034-plugins-config-volume" pod "ee57b283-ffc3-4310-9113-6568a46d0034" (UID: "ee57b283-ffc3-4310-9113-6568a46d0034"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:48:25.498933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.498909 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee57b283-ffc3-4310-9113-6568a46d0034-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:25.499615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.499586 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07dac19e-1503-410b-8eab-6905bb48ec0b-plugins-config-volume" pod "07dac19e-1503-410b-8eab-6905bb48ec0b" (UID: "07dac19e-1503-410b-8eab-6905bb48ec0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:48:25.500790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.500764 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee57b283-ffc3-4310-9113-6568a46d0034-kube-api-access-2rww9" pod "ee57b283-ffc3-4310-9113-6568a46d0034" (UID: "ee57b283-ffc3-4310-9113-6568a46d0034"). InnerVolumeSpecName "kube-api-access-2rww9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:48:25.501430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.501403 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07dac19e-1503-410b-8eab-6905bb48ec0b-kube-api-access-4s6ct" pod "07dac19e-1503-410b-8eab-6905bb48ec0b" (UID: "07dac19e-1503-410b-8eab-6905bb48ec0b"). InnerVolumeSpecName "kube-api-access-4s6ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:48:25.600662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.600577 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dac19e-1503-410b-8eab-6905bb48ec0b-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:25.600662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.600610 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rww9\" (UniqueName: \"kubernetes.io/projected/ee57b283-ffc3-4310-9113-6568a46d0034-kube-api-access-2rww9\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:25.600662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.600623 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s6ct\" (UniqueName: \"kubernetes.io/projected/07dac19e-1503-410b-8eab-6905bb48ec0b-kube-api-access-4s6ct\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:25.607100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.607068 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:25.681172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.681132 5591 generic.go:356] "Generic (PLEG): container finished" podID="ee57b283-ffc3-4310-9113-6568a46d0034" containerID="3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39" exitCode=0 Nov 02 14:48:25.681334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.681176 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" Nov 02 14:48:25.681334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.681215 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" event={"ID":"ee57b283-ffc3-4310-9113-6568a46d0034","Type":"ContainerDied","Data":"3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39"} Nov 02 14:48:25.681334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.681268 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4" event={"ID":"ee57b283-ffc3-4310-9113-6568a46d0034","Type":"ContainerDied","Data":"a1406b104a7bf0776cf5c916e9528e08671b9a476240febdf94797486636d700"} Nov 02 14:48:25.681334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.681298 5591 scope.go:122] "RemoveContainer" containerID="3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39" Nov 02 14:48:25.686362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.683024 5591 generic.go:356] "Generic (PLEG): container finished" podID="07dac19e-1503-410b-8eab-6905bb48ec0b" containerID="b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107" exitCode=0 Nov 02 14:48:25.686362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.683065 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" Nov 02 14:48:25.686362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.683062 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" event={"ID":"07dac19e-1503-410b-8eab-6905bb48ec0b","Type":"ContainerDied","Data":"b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107"} Nov 02 14:48:25.686362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.683196 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5" event={"ID":"07dac19e-1503-410b-8eab-6905bb48ec0b","Type":"ContainerDied","Data":"d4fd3b92c1cff6d8aa41437e87fbbd746df3b44b8cb008ae1c6258e4cd9832fd"} Nov 02 14:48:25.697884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.694934 5591 scope.go:122] "RemoveContainer" containerID="3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39" Nov 02 14:48:25.697884 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:25.695317 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39\": container with ID starting with 3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39 not found: ID does not exist" containerID="3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39" Nov 02 14:48:25.697884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.695351 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39"} err="failed to get container status \"3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39\": rpc error: code = NotFound desc = could not find container \"3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39\": container with ID starting with 3d829f1d3a37e22f2504e88cf9fef99ad9287fd9e289355c3996f5af8d3fcf39 not found: ID does not exist" Nov 02 14:48:25.697884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.695373 5591 scope.go:122] "RemoveContainer" containerID="b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107" Nov 02 14:48:25.709786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.709755 5591 scope.go:122] "RemoveContainer" containerID="b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107" Nov 02 14:48:25.710129 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:25.710102 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107\": container with ID starting with b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107 not found: ID does not exist" containerID="b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107" Nov 02 14:48:25.710223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.710136 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107"} err="failed to get container status \"b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107\": rpc error: code = NotFound desc = could not find container \"b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107\": container with ID starting with b156924d8348be05e2c3713f379da9b626e07b12f646df3d5a04716417d0f107 not found: ID does not exist" Nov 02 14:48:25.755878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.755814 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5"] Nov 02 14:48:25.761938 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.761906 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea3e8f-6c8c559f99-dfcq5"] Nov 02 14:48:25.797227 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.797192 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4"] Nov 02 14:48:25.810887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.810854 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea3e8f-5774f4f67-cpbk4"] Nov 02 14:48:25.957162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:25.957123 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:26.160693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:26.160658 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:26.681622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:26.681587 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:27.433682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:27.433649 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07dac19e-1503-410b-8eab-6905bb48ec0b" path="/var/lib/kubelet/pods/07dac19e-1503-410b-8eab-6905bb48ec0b/volumes" Nov 02 14:48:27.434149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:27.434104 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee57b283-ffc3-4310-9113-6568a46d0034" path="/var/lib/kubelet/pods/ee57b283-ffc3-4310-9113-6568a46d0034/volumes" Nov 02 14:48:27.799767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:27.799670 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:28.394633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:28.394589 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:30.071007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:30.048229 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:31.075112 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:31.075072 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:48:31.433558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:31.433520 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5"] Nov 02 14:48:31.591129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:31.591088 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:31.820516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:31.820412 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:31.984192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:31.984154 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:32.343913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:32.343880 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:33.662437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:33.662395 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:35.281409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:35.281370 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:36.435135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:36.435090 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" podUID="ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" containerName="payload-pre-processing" containerID="cri-o://db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f" gracePeriod=25 Nov 02 14:48:36.861853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:36.861704 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:38.737136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:38.737091 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" podUID="ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:48:38.963532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:38.963496 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:48:40.100904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.100853 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" podUID="79dea7ff-8473-4fb6-8325-294bcec1299c" containerName="main" containerID="cri-o://21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b" gracePeriod=45 Nov 02 14:48:40.481435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.481404 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:48:40.495565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.495536 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:40.641197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641163 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-model-cache\") pod \"79dea7ff-8473-4fb6-8325-294bcec1299c\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " Nov 02 14:48:40.641375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641328 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dea7ff-8473-4fb6-8325-294bcec1299c-kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/79dea7ff-8473-4fb6-8325-294bcec1299c-kube-api-access-gbplv\") pod \"79dea7ff-8473-4fb6-8325-294bcec1299c\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " Nov 02 14:48:40.641436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641371 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-home\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-home\") pod \"79dea7ff-8473-4fb6-8325-294bcec1299c\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " Nov 02 14:48:40.641436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641430 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-tmp-dir\") pod \"79dea7ff-8473-4fb6-8325-294bcec1299c\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " Nov 02 14:48:40.641554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641433 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-model-cache" pod "79dea7ff-8473-4fb6-8325-294bcec1299c" (UID: "79dea7ff-8473-4fb6-8325-294bcec1299c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:40.641554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641452 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79dea7ff-8473-4fb6-8325-294bcec1299c-tls-certs\" (UniqueName: \"kubernetes.io/secret/79dea7ff-8473-4fb6-8325-294bcec1299c-tls-certs\") pod \"79dea7ff-8473-4fb6-8325-294bcec1299c\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " Nov 02 14:48:40.641554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641489 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-dshm\") pod \"79dea7ff-8473-4fb6-8325-294bcec1299c\" (UID: \"79dea7ff-8473-4fb6-8325-294bcec1299c\") " Nov 02 14:48:40.641753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641730 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-tmp-dir" pod "79dea7ff-8473-4fb6-8325-294bcec1299c" (UID: "79dea7ff-8473-4fb6-8325-294bcec1299c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:40.641828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.641745 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-home" pod "79dea7ff-8473-4fb6-8325-294bcec1299c" (UID: "79dea7ff-8473-4fb6-8325-294bcec1299c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:40.642220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.642200 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:40.642322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.642222 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:40.642322 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.642236 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:40.643716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.643693 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-dshm" pod "79dea7ff-8473-4fb6-8325-294bcec1299c" (UID: "79dea7ff-8473-4fb6-8325-294bcec1299c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:40.643820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.643762 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79dea7ff-8473-4fb6-8325-294bcec1299c-tls-certs" pod "79dea7ff-8473-4fb6-8325-294bcec1299c" (UID: "79dea7ff-8473-4fb6-8325-294bcec1299c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:48:40.643820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.643777 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dea7ff-8473-4fb6-8325-294bcec1299c-kube-api-access-gbplv" pod "79dea7ff-8473-4fb6-8325-294bcec1299c" (UID: "79dea7ff-8473-4fb6-8325-294bcec1299c"). InnerVolumeSpecName "kube-api-access-gbplv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:48:40.743265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.743233 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79dea7ff-8473-4fb6-8325-294bcec1299c-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:40.743265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.743260 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dea7ff-8473-4fb6-8325-294bcec1299c-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:40.743265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.743269 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/79dea7ff-8473-4fb6-8325-294bcec1299c-kube-api-access-gbplv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:40.813767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.813729 5591 generic.go:356] "Generic (PLEG): container finished" podID="79dea7ff-8473-4fb6-8325-294bcec1299c" containerID="21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b" exitCode=0 Nov 02 14:48:40.813958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.813769 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" Nov 02 14:48:40.813958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.813785 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" event={"ID":"79dea7ff-8473-4fb6-8325-294bcec1299c","Type":"ContainerDied","Data":"21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b"} Nov 02 14:48:40.813958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.813823 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf" event={"ID":"79dea7ff-8473-4fb6-8325-294bcec1299c","Type":"ContainerDied","Data":"a1b64cb715be4caaa7d3ff460d924fef22309ff3de392dc4966b1ef6f175a3f6"} Nov 02 14:48:40.813958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.813840 5591 scope.go:122] "RemoveContainer" containerID="21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b" Nov 02 14:48:40.824747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.824719 5591 scope.go:122] "RemoveContainer" containerID="21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b" Nov 02 14:48:40.825074 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:40.825052 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b\": container with ID starting with 21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b not found: ID does not exist" containerID="21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b" Nov 02 14:48:40.825139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.825083 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b"} err="failed to get container status \"21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b\": rpc error: code = NotFound desc = could not find container \"21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b\": container with ID starting with 21f1312eb95e2c3b3f3354d2f976dbec5ca220741b58551218b0560655f8af5b not found: ID does not exist" Nov 02 14:48:40.871460 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.871425 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf"] Nov 02 14:48:40.877285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:40.877257 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-80133f/test-model-80133f-kserve-848cdc4b77-lppkf"] Nov 02 14:48:41.437187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:41.437154 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79dea7ff-8473-4fb6-8325-294bcec1299c" path="/var/lib/kubelet/pods/79dea7ff-8473-4fb6-8325-294bcec1299c/volumes" Nov 02 14:48:41.944817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:41.944664 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:43.603977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:43.603931 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:44.588888 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:44.588841 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:45.064575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:45.064545 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:45.872179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:45.872138 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:47.543996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:47.543939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:48.736801 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:48.736759 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" podUID="ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:48:49.100056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:49.091133 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:50.858434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:50.858395 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:51.166441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:51.165857 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:51.353201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:51.353166 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:51.723479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:51.723441 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:51.903037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:51.902969 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:52.204464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:52.204424 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w"] Nov 02 14:48:52.204767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:52.204733 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" containerID="cri-o://e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21" gracePeriod=30 Nov 02 14:48:52.333918 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:52.333884 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:52.773178 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:52.773067 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:53.508709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:53.508550 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:54.054005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:54.051706 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:54.410816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:54.410780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:54.653673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:54.653627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:56.299836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.299790 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:56.714022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.713970 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:48:56.798576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.798532 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-plugins-config-volume\") pod \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\" (UID: \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\") " Nov 02 14:48:56.798785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.798593 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-kube-api-access-fjj7g\" (UniqueName: \"kubernetes.io/projected/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-kube-api-access-fjj7g\") pod \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\" (UID: \"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b\") " Nov 02 14:48:56.799016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.798958 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-plugins-config-volume" pod "ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" (UID: "ea8e0f08-2dba-4a35-9c8a-07fa40182d4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:48:56.799400 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.799367 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:56.800895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.800862 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-kube-api-access-fjj7g" pod "ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" (UID: "ea8e0f08-2dba-4a35-9c8a-07fa40182d4b"). InnerVolumeSpecName "kube-api-access-fjj7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:48:56.900915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.900878 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjj7g\" (UniqueName: \"kubernetes.io/projected/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b-kube-api-access-fjj7g\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:56.960822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.960782 5591 generic.go:356] "Generic (PLEG): container finished" podID="ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" containerID="db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f" exitCode=0 Nov 02 14:48:56.961038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.960826 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" Nov 02 14:48:56.961038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.960866 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" event={"ID":"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b","Type":"ContainerDied","Data":"db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f"} Nov 02 14:48:56.961038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.960919 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5" event={"ID":"ea8e0f08-2dba-4a35-9c8a-07fa40182d4b","Type":"ContainerDied","Data":"b5700d70ed2793d999a3e5769e6aa1b83eb3367d264e68a7f6af72f75e31414f"} Nov 02 14:48:56.961038 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.960941 5591 scope.go:122] "RemoveContainer" containerID="db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f" Nov 02 14:48:56.971946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.971925 5591 scope.go:122] "RemoveContainer" containerID="db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f" Nov 02 14:48:56.972253 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:56.972232 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f\": container with ID starting with db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f not found: ID does not exist" containerID="db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f" Nov 02 14:48:56.972305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:56.972261 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f"} err="failed to get container status \"db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f\": rpc error: code = NotFound desc = could not find container \"db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f\": container with ID starting with db2fd33b6e1cb16bdb3a1808a3f17393a95783c6d35cc94e8ff529a107a4a12f not found: ID does not exist" Nov 02 14:48:57.016289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.016252 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5"] Nov 02 14:48:57.022498 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.022474 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8f647-5588fb475-8m6j5"] Nov 02 14:48:57.434835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.434803 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" path="/var/lib/kubelet/pods/ea8e0f08-2dba-4a35-9c8a-07fa40182d4b/volumes" Nov 02 14:48:57.477599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.477534 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:48:57.610455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.610410 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91593d10-d549-4bf5-a212-0a2afb872125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91593d10-d549-4bf5-a212-0a2afb872125-istiod-ca-cert\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.610705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.610578 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-istio-token\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-istio-token\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.610705 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.610618 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-kube-api-access-f2wtz\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-kube-api-access-f2wtz\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.610863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.610825 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91593d10-d549-4bf5-a212-0a2afb872125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91593d10-d549-4bf5-a212-0a2afb872125-istio-podinfo\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.610965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.610950 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-certs\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.611136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.610832 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91593d10-d549-4bf5-a212-0a2afb872125-istiod-ca-cert" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:48:57.611136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.611033 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-socket\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.611448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.611360 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-socket" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:57.611448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.611439 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-certs" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:57.612222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.612082 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-data\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.612222 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.612190 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-data" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:57.612409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.612229 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-envoy\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.612578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.612558 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-credential-socket\") pod \"91593d10-d549-4bf5-a212-0a2afb872125\" (UID: \"91593d10-d549-4bf5-a212-0a2afb872125\") " Nov 02 14:48:57.612916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.612880 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-credential-socket" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:57.614582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.613517 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91593d10-d549-4bf5-a212-0a2afb872125-istio-podinfo" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:48:57.614582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614380 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-istio-token" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:48:57.614582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614466 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-kube-api-access-f2wtz" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "kube-api-access-f2wtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:48:57.614825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614736 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91593d10-d549-4bf5-a212-0a2afb872125-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.614825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614757 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.614825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614770 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2wtz\" (UniqueName: \"kubernetes.io/projected/91593d10-d549-4bf5-a212-0a2afb872125-kube-api-access-f2wtz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.614825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614785 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91593d10-d549-4bf5-a212-0a2afb872125-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.614825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614798 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.614825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614813 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.614825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614821 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.615245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.614829 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.615881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.615855 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-envoy" pod "91593d10-d549-4bf5-a212-0a2afb872125" (UID: "91593d10-d549-4bf5-a212-0a2afb872125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:48:57.715741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.715702 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91593d10-d549-4bf5-a212-0a2afb872125-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:48:57.969239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.969202 5591 generic.go:356] "Generic (PLEG): container finished" podID="91593d10-d549-4bf5-a212-0a2afb872125" containerID="e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21" exitCode=0 Nov 02 14:48:57.969456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.969255 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" Nov 02 14:48:57.969456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.969299 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" event={"ID":"91593d10-d549-4bf5-a212-0a2afb872125","Type":"ContainerDied","Data":"e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21"} Nov 02 14:48:57.969456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.969351 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w" event={"ID":"91593d10-d549-4bf5-a212-0a2afb872125","Type":"ContainerDied","Data":"a56d91f59ff20ef75838e7fd62692c7a0f07cefa116ca458d098c8afd9fb2dfa"} Nov 02 14:48:57.969456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.969381 5591 scope.go:122] "RemoveContainer" containerID="e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21" Nov 02 14:48:57.981182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.981162 5591 scope.go:122] "RemoveContainer" containerID="de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75" Nov 02 14:48:57.991022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.991000 5591 scope.go:122] "RemoveContainer" containerID="e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21" Nov 02 14:48:57.991285 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:57.991263 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21\": container with ID starting with e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21 not found: ID does not exist" containerID="e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21" Nov 02 14:48:57.991361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.991294 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21"} err="failed to get container status \"e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21\": rpc error: code = NotFound desc = could not find container \"e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21\": container with ID starting with e7e3695402f5858f2308818c340905586b683f8ee561031537d4084933b92f21 not found: ID does not exist" Nov 02 14:48:57.991361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.991311 5591 scope.go:122] "RemoveContainer" containerID="de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75" Nov 02 14:48:57.991544 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:48:57.991528 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75\": container with ID starting with de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75 not found: ID does not exist" containerID="de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75" Nov 02 14:48:57.991586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:57.991547 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75"} err="failed to get container status \"de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75\": rpc error: code = NotFound desc = could not find container \"de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75\": container with ID starting with de8028b085e8185e185012b451987277410af8b039635ed587e530372bb9bb75 not found: ID does not exist" Nov 02 14:48:58.026819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:58.026780 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w"] Nov 02 14:48:58.038444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:58.038413 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-84b8w"] Nov 02 14:48:58.073911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:58.073830 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:48:59.432539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:59.432506 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91593d10-d549-4bf5-a212-0a2afb872125" path="/var/lib/kubelet/pods/91593d10-d549-4bf5-a212-0a2afb872125/volumes" Nov 02 14:48:59.793100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:48:59.793010 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:01.074589 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:49:01.074549 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:49:01.394220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:01.392174 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:03.079970 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:03.079631 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:04.687947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:04.687908 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:06.384744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:06.384680 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:08.119348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:08.119288 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:09.870627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:09.870580 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:10.248325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:10.248288 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:10.465726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:10.465690 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:10.667973 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:10.667940 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:11.870005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:11.866662 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:12.137171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.137136 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:12.216927 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.216898 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:49:12.280230 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.280194 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:49:12.281135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281104 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j"] Nov 02 14:49:12.281873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281849 5591 cpu_manager.go:414] "containerMap: removing container" podUID="79dea7ff-8473-4fb6-8325-294bcec1299c" containerName="main" Nov 02 14:49:12.281873 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281874 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79dea7ff-8473-4fb6-8325-294bcec1299c" containerName="main" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281894 5591 cpu_manager.go:414] "containerMap: removing container" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281900 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281933 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c782b989-1334-4b2d-a7dd-1080b0951ea7" containerName="main" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281938 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c782b989-1334-4b2d-a7dd-1080b0951ea7" containerName="main" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281953 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ee57b283-ffc3-4310-9113-6568a46d0034" containerName="payload-pre-processing" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281958 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee57b283-ffc3-4310-9113-6568a46d0034" containerName="payload-pre-processing" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281964 5591 cpu_manager.go:414] "containerMap: removing container" podUID="07dac19e-1503-410b-8eab-6905bb48ec0b" containerName="payload-processing" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281969 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07dac19e-1503-410b-8eab-6905bb48ec0b" containerName="payload-processing" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.281997 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" containerName="payload-pre-processing" Nov 02 14:49:12.282055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.282003 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" containerName="payload-pre-processing" Nov 02 14:49:12.282470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.282117 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c782b989-1334-4b2d-a7dd-1080b0951ea7" containerName="main" Nov 02 14:49:12.282470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.282128 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" Nov 02 14:49:12.282470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.282134 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee57b283-ffc3-4310-9113-6568a46d0034" containerName="payload-pre-processing" Nov 02 14:49:12.282470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.282141 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="07dac19e-1503-410b-8eab-6905bb48ec0b" containerName="payload-processing" Nov 02 14:49:12.282470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.282153 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8e0f08-2dba-4a35-9c8a-07fa40182d4b" containerName="payload-pre-processing" Nov 02 14:49:12.282470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.282165 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="79dea7ff-8473-4fb6-8325-294bcec1299c" containerName="main" Nov 02 14:49:12.282470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.282177 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" Nov 02 14:49:12.287790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.287773 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:12.321214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.321158 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbq9\" (UniqueName: \"kubernetes.io/projected/17d10963-ca4b-4b84-ae31-a8b9981f78b3-kube-api-access-wzbq9\") pod \"payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j\" (UID: \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:12.321473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.321448 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d10963-ca4b-4b84-ae31-a8b9981f78b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j\" (UID: \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:12.325061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.325031 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e16d1c0e-dockercfg-gnqq9\"" Nov 02 14:49:12.325252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.325101 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e16d1c0e\"" Nov 02 14:49:12.355047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.329910 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j"] Nov 02 14:49:12.355047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.343574 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:12.442327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.442231 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbq9\" (UniqueName: \"kubernetes.io/projected/17d10963-ca4b-4b84-ae31-a8b9981f78b3-kube-api-access-wzbq9\") pod \"payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j\" (UID: \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:12.443160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.443103 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d10963-ca4b-4b84-ae31-a8b9981f78b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j\" (UID: \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:12.445224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.444854 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d10963-ca4b-4b84-ae31-a8b9981f78b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j\" (UID: \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:12.460043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.456048 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82"] Nov 02 14:49:12.460043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.457201 5591 cpu_manager.go:414] "containerMap: removing container" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" Nov 02 14:49:12.460043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.457223 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91593d10-d549-4bf5-a212-0a2afb872125" containerName="istio-proxy" Nov 02 14:49:12.473726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.464856 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:12.473726 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.465161 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82"] Nov 02 14:49:12.525152 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.525082 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbq9\" (UniqueName: \"kubernetes.io/projected/17d10963-ca4b-4b84-ae31-a8b9981f78b3-kube-api-access-wzbq9\") pod \"payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j\" (UID: \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:12.549047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.545201 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2lz5\" (UniqueName: \"kubernetes.io/projected/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-kube-api-access-l2lz5\") pod \"payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82\" (UID: \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:12.549047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.545599 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82\" (UID: \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:12.602459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.602421 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:12.647754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.647719 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2lz5\" (UniqueName: \"kubernetes.io/projected/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-kube-api-access-l2lz5\") pod \"payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82\" (UID: \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:12.647949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.647925 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82\" (UID: \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:12.648685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.648662 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82\" (UID: \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:12.685786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.685747 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2lz5\" (UniqueName: \"kubernetes.io/projected/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-kube-api-access-l2lz5\") pod \"payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82\" (UID: \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:12.864311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.864239 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:12.913632 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:49:12.913595 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d10963_ca4b_4b84_ae31_a8b9981f78b3.slice/crio-ade3e36ca208b941547980f28959bf199f4ceeb353e169319087ddfe19ca88af WatchSource:0}: Error finding container ade3e36ca208b941547980f28959bf199f4ceeb353e169319087ddfe19ca88af: Status 404 returned error can't find the container with id ade3e36ca208b941547980f28959bf199f4ceeb353e169319087ddfe19ca88af Nov 02 14:49:12.913966 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:12.913922 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j"] Nov 02 14:49:13.078507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:13.078465 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" event={"ID":"17d10963-ca4b-4b84-ae31-a8b9981f78b3","Type":"ContainerStarted","Data":"d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179"} Nov 02 14:49:13.078685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:13.078526 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" event={"ID":"17d10963-ca4b-4b84-ae31-a8b9981f78b3","Type":"ContainerStarted","Data":"ade3e36ca208b941547980f28959bf199f4ceeb353e169319087ddfe19ca88af"} Nov 02 14:49:13.078685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:13.078557 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:13.142329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:13.135700 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" podStartSLOduration=1.135668076 podStartE2EDuration="1.135668076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:13.130083389 +0000 UTC m=+3901.855705159" watchObservedRunningTime="2026-11-02 14:49:13.135668076 +0000 UTC m=+3901.861289847" Nov 02 14:49:13.175540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:13.175512 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82"] Nov 02 14:49:13.176754 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:49:13.176726 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6b9d59_c7f6_4f43_b2b7_85137885ec3a.slice/crio-138a69463a5fa6ff4c649571a889e415e7ab2c678c14f7bc78b6c7896932b767 WatchSource:0}: Error finding container 138a69463a5fa6ff4c649571a889e415e7ab2c678c14f7bc78b6c7896932b767: Status 404 returned error can't find the container with id 138a69463a5fa6ff4c649571a889e415e7ab2c678c14f7bc78b6c7896932b767 Nov 02 14:49:13.769009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:13.760549 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:14.087866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:14.087764 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" event={"ID":"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a","Type":"ContainerStarted","Data":"de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055"} Nov 02 14:49:14.087866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:14.087822 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" event={"ID":"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a","Type":"ContainerStarted","Data":"138a69463a5fa6ff4c649571a889e415e7ab2c678c14f7bc78b6c7896932b767"} Nov 02 14:49:14.088370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:14.087975 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:14.130956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:14.130901 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" podStartSLOduration=2.130887978 podStartE2EDuration="2.130887978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:14.130454214 +0000 UTC m=+3902.856075988" watchObservedRunningTime="2026-11-02 14:49:14.130887978 +0000 UTC m=+3902.856509768" Nov 02 14:49:15.496117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:15.496068 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:17.030348 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:17.030315 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:17.865412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:17.864078 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:18.544811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:18.544774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:18.738255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:18.738220 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:19.681251 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:19.681205 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:19.891229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:19.891179 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:20.103820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:20.103701 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:21.655712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:21.655658 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:23.253233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:23.252225 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:23.729714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:23.729681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:23.917736 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:23.917676 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:24.091022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:24.090931 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:49:24.529629 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:24.529595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:24.978369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:24.978333 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:25.141856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:25.141822 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:49:25.377472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:25.374108 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:25.835625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:25.835589 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:27.145005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:27.144785 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:27.799261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:27.799223 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:28.810008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:28.807650 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:30.587651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:30.587605 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:31.075561 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:49:31.075526 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:49:31.133043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:31.133006 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g"] Nov 02 14:49:31.167268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:31.167232 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69"] Nov 02 14:49:31.317192 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:31.317161 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:31.483284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:31.483240 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:32.183297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:32.183255 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:32.704063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:32.704026 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:33.127962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:33.127861 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:34.505766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:34.505730 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:35.990614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:35.990569 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:49:36.134062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.134021 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" podUID="a8cef1df-75f5-4a87-970e-257e87108d9b" containerName="payload-processing" containerID="cri-o://04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9" gracePeriod=25 Nov 02 14:49:36.167664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.167628 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" podUID="53af4c44-b710-4d7f-a167-d86391499b3c" containerName="payload-pre-processing" containerID="cri-o://bcc5c2182ea113bcaaf541ffe610b817703950150c5c7ab53417a550e97588a6" gracePeriod=25 Nov 02 14:49:36.268060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.263207 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:36.453070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.453043 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:49:36.590777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.590688 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cef1df-75f5-4a87-970e-257e87108d9b-kube-api-access-xbhgf\" (UniqueName: \"kubernetes.io/projected/a8cef1df-75f5-4a87-970e-257e87108d9b-kube-api-access-xbhgf\") pod \"a8cef1df-75f5-4a87-970e-257e87108d9b\" (UID: \"a8cef1df-75f5-4a87-970e-257e87108d9b\") " Nov 02 14:49:36.590777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.590738 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cef1df-75f5-4a87-970e-257e87108d9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cef1df-75f5-4a87-970e-257e87108d9b-plugins-config-volume\") pod \"a8cef1df-75f5-4a87-970e-257e87108d9b\" (UID: \"a8cef1df-75f5-4a87-970e-257e87108d9b\") " Nov 02 14:49:36.591274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.591247 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cef1df-75f5-4a87-970e-257e87108d9b-plugins-config-volume" pod "a8cef1df-75f5-4a87-970e-257e87108d9b" (UID: "a8cef1df-75f5-4a87-970e-257e87108d9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:49:36.592902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.592877 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cef1df-75f5-4a87-970e-257e87108d9b-kube-api-access-xbhgf" pod "a8cef1df-75f5-4a87-970e-257e87108d9b" (UID: "a8cef1df-75f5-4a87-970e-257e87108d9b"). InnerVolumeSpecName "kube-api-access-xbhgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:49:36.692604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.692566 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhgf\" (UniqueName: \"kubernetes.io/projected/a8cef1df-75f5-4a87-970e-257e87108d9b-kube-api-access-xbhgf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:49:36.692604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.692598 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cef1df-75f5-4a87-970e-257e87108d9b-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:49:36.780579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:36.780536 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" podUID="53af4c44-b710-4d7f-a167-d86391499b3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:49:37.283025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.282964 5591 generic.go:356] "Generic (PLEG): container finished" podID="a8cef1df-75f5-4a87-970e-257e87108d9b" containerID="04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9" exitCode=0 Nov 02 14:49:37.283025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.283011 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" event={"ID":"a8cef1df-75f5-4a87-970e-257e87108d9b","Type":"ContainerDied","Data":"04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9"} Nov 02 14:49:37.283606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.283040 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" Nov 02 14:49:37.283606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.283080 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g" event={"ID":"a8cef1df-75f5-4a87-970e-257e87108d9b","Type":"ContainerDied","Data":"9bd0810b3b6083e4d3136904ba266addc0e2124af486a6d8b78d8e7ca9efabcf"} Nov 02 14:49:37.283606 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.283114 5591 scope.go:122] "RemoveContainer" containerID="04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9" Nov 02 14:49:37.297134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.297100 5591 scope.go:122] "RemoveContainer" containerID="04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9" Nov 02 14:49:37.297645 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:49:37.297607 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9\": container with ID starting with 04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9 not found: ID does not exist" containerID="04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9" Nov 02 14:49:37.297766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.297656 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9"} err="failed to get container status \"04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9\": rpc error: code = NotFound desc = could not find container \"04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9\": container with ID starting with 04b56f329e0e33c1ee684cb1fa2a8ef02d01475712da92426314f3353bce7bf9 not found: ID does not exist" Nov 02 14:49:37.350505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.350458 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g"] Nov 02 14:49:37.363315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.363273 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80133f-76b6f9f977-lhh6g"] Nov 02 14:49:37.435169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:37.435133 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cef1df-75f5-4a87-970e-257e87108d9b" path="/var/lib/kubelet/pods/a8cef1df-75f5-4a87-970e-257e87108d9b/volumes" Nov 02 14:49:38.066093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:38.066055 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:38.758008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:38.757494 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:39.411810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:39.408967 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:39.932533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:39.932495 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:41.683425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:41.683379 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:41.882074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:41.882038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:42.058946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:42.058841 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:43.510565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:43.510529 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:45.262822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:45.262782 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:46.780578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:46.780537 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" podUID="53af4c44-b710-4d7f-a167-d86391499b3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:49:46.980439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:46.980400 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:48.449149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:48.449110 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:48.663450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:48.663400 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:48.860656 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:48.860548 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:50.384139 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.384107 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk"] Nov 02 14:49:50.385079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.385058 5591 cpu_manager.go:414] "containerMap: removing container" podUID="a8cef1df-75f5-4a87-970e-257e87108d9b" containerName="payload-processing" Nov 02 14:49:50.385079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.385080 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cef1df-75f5-4a87-970e-257e87108d9b" containerName="payload-processing" Nov 02 14:49:50.385292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.385277 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cef1df-75f5-4a87-970e-257e87108d9b" containerName="payload-processing" Nov 02 14:49:50.389554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.389519 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.409164 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.409133 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-960a68\"/\"sub-test-model-960a68-kserve-self-signed-certs\"" Nov 02 14:49:50.422300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.422271 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-960a68\"/\"kube-root-ca.crt\"" Nov 02 14:49:50.422457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.422342 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-960a68\"/\"default-dockercfg-bhkht\"" Nov 02 14:49:50.422457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.422432 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-960a68\"/\"openshift-service-ca.crt\"" Nov 02 14:49:50.433672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.433640 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk"] Nov 02 14:49:50.436368 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.436339 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tmp-dir\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.436506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.436469 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-model-cache\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.436575 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.436561 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgrz4\" (UniqueName: \"kubernetes.io/projected/39540df0-9900-4ded-8dd2-d21ed94e1dbc-kube-api-access-bgrz4\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.436720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.436694 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-home\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.437165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.437140 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-dshm\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.437263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.437194 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tls-certs\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.532332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.532297 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:50.538770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.538742 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tmp-dir\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.538912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.538780 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-model-cache\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.538912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.538822 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgrz4\" (UniqueName: \"kubernetes.io/projected/39540df0-9900-4ded-8dd2-d21ed94e1dbc-kube-api-access-bgrz4\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.538912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.538897 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-home\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.539120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.539018 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-dshm\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.539120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.539066 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tls-certs\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.539284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.539267 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-model-cache\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.539353 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.539293 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tmp-dir\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.539425 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.539368 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-home\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.541267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.541245 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-dshm\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.541655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.541633 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tls-certs\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.568344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.568314 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgrz4\" (UniqueName: \"kubernetes.io/projected/39540df0-9900-4ded-8dd2-d21ed94e1dbc-kube-api-access-bgrz4\") pod \"sub-test-model-960a68-kserve-648f7c5dc7-vqnzk\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.716787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.716433 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:50.739487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:50.739313 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:51.083445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:51.083412 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk"] Nov 02 14:49:51.084021 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:49:51.083978 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39540df0_9900_4ded_8dd2_d21ed94e1dbc.slice/crio-76e5ea3e9bcb69159109f5c3ea622bbd4ba285519daaeb46986f390c85def46e WatchSource:0}: Error finding container 76e5ea3e9bcb69159109f5c3ea622bbd4ba285519daaeb46986f390c85def46e: Status 404 returned error can't find the container with id 76e5ea3e9bcb69159109f5c3ea622bbd4ba285519daaeb46986f390c85def46e Nov 02 14:49:51.414552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:51.414512 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" event={"ID":"39540df0-9900-4ded-8dd2-d21ed94e1dbc","Type":"ContainerStarted","Data":"4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c"} Nov 02 14:49:51.414991 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:51.414565 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" event={"ID":"39540df0-9900-4ded-8dd2-d21ed94e1dbc","Type":"ContainerStarted","Data":"76e5ea3e9bcb69159109f5c3ea622bbd4ba285519daaeb46986f390c85def46e"} Nov 02 14:49:51.463143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:51.463086 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" podStartSLOduration=1.463073788 podStartE2EDuration="1.463073788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:51.461389337 +0000 UTC m=+3940.187011107" watchObservedRunningTime="2026-11-02 14:49:51.463073788 +0000 UTC m=+3940.188695595" Nov 02 14:49:51.718061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:51.717963 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:49:52.491461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:52.491427 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:54.198520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:54.198487 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:56.182793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.182745 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:56.456374 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.456330 5591 generic.go:356] "Generic (PLEG): container finished" podID="53af4c44-b710-4d7f-a167-d86391499b3c" containerID="bcc5c2182ea113bcaaf541ffe610b817703950150c5c7ab53417a550e97588a6" exitCode=0 Nov 02 14:49:56.456525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.456397 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" event={"ID":"53af4c44-b710-4d7f-a167-d86391499b3c","Type":"ContainerDied","Data":"bcc5c2182ea113bcaaf541ffe610b817703950150c5c7ab53417a550e97588a6"} Nov 02 14:49:56.483989 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.483951 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:49:56.526464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.526425 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53af4c44-b710-4d7f-a167-d86391499b3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53af4c44-b710-4d7f-a167-d86391499b3c-plugins-config-volume\") pod \"53af4c44-b710-4d7f-a167-d86391499b3c\" (UID: \"53af4c44-b710-4d7f-a167-d86391499b3c\") " Nov 02 14:49:56.526637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.526604 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53af4c44-b710-4d7f-a167-d86391499b3c-kube-api-access-h7vch\" (UniqueName: \"kubernetes.io/projected/53af4c44-b710-4d7f-a167-d86391499b3c-kube-api-access-h7vch\") pod \"53af4c44-b710-4d7f-a167-d86391499b3c\" (UID: \"53af4c44-b710-4d7f-a167-d86391499b3c\") " Nov 02 14:49:56.526943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.526914 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53af4c44-b710-4d7f-a167-d86391499b3c-plugins-config-volume" pod "53af4c44-b710-4d7f-a167-d86391499b3c" (UID: "53af4c44-b710-4d7f-a167-d86391499b3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:49:56.527959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.527931 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53af4c44-b710-4d7f-a167-d86391499b3c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:49:56.533086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.529143 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53af4c44-b710-4d7f-a167-d86391499b3c-kube-api-access-h7vch" pod "53af4c44-b710-4d7f-a167-d86391499b3c" (UID: "53af4c44-b710-4d7f-a167-d86391499b3c"). InnerVolumeSpecName "kube-api-access-h7vch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:49:56.629485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:56.629443 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7vch\" (UniqueName: \"kubernetes.io/projected/53af4c44-b710-4d7f-a167-d86391499b3c-kube-api-access-h7vch\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:49:57.464643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:57.464608 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" event={"ID":"53af4c44-b710-4d7f-a167-d86391499b3c","Type":"ContainerDied","Data":"def64c18f39e879fa4c1a8008391f1fa8cc78c499086697b050511662e53e315"} Nov 02 14:49:57.464643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:57.464625 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69" Nov 02 14:49:57.465159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:57.464678 5591 scope.go:122] "RemoveContainer" containerID="bcc5c2182ea113bcaaf541ffe610b817703950150c5c7ab53417a550e97588a6" Nov 02 14:49:57.524749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:57.524709 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69"] Nov 02 14:49:57.537978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:57.537946 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80133f-5648c95c9-mlg69"] Nov 02 14:49:58.147600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:58.147564 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:49:59.433900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:59.433864 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53af4c44-b710-4d7f-a167-d86391499b3c" path="/var/lib/kubelet/pods/53af4c44-b710-4d7f-a167-d86391499b3c/volumes" Nov 02 14:49:59.970890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:59.970856 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x"] Nov 02 14:49:59.971674 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:59.971655 5591 cpu_manager.go:414] "containerMap: removing container" podUID="53af4c44-b710-4d7f-a167-d86391499b3c" containerName="payload-pre-processing" Nov 02 14:49:59.971738 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:59.971676 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53af4c44-b710-4d7f-a167-d86391499b3c" containerName="payload-pre-processing" Nov 02 14:49:59.971822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:59.971811 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="53af4c44-b710-4d7f-a167-d86391499b3c" containerName="payload-pre-processing" Nov 02 14:49:59.976798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:49:59.976780 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.002585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.002553 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x"] Nov 02 14:50:00.011977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.011941 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e16d1c0e-serving-cert\"" Nov 02 14:50:00.077623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.077587 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.077850 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.077797 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-metrics-certs\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.077850 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.077844 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwcl\" (UniqueName: \"kubernetes.io/projected/36dd4024-c8de-4906-a1c6-cc0a0e757574-kube-api-access-jgwcl\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.078067 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.078043 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dd4024-c8de-4906-a1c6-cc0a0e757574-openshift-service-ca\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.169391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.169352 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:00.179435 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.179398 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-metrics-certs\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.179603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.179458 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwcl\" (UniqueName: \"kubernetes.io/projected/36dd4024-c8de-4906-a1c6-cc0a0e757574-kube-api-access-jgwcl\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.179603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.179543 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dd4024-c8de-4906-a1c6-cc0a0e757574-openshift-service-ca\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.179701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.179629 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.179764 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:50:00.179747 5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-serving-cert: secret "maas-api-e2e-mt-e16d1c0e-serving-cert" not found Nov 02 14:50:00.179861 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:50:00.179812 5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls podName:36dd4024-c8de-4906-a1c6-cc0a0e757574 nodeName:}" failed. No retries permitted until 2026-11-02 14:50:00.679795818 +0000 UTC m=+3949.405417586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls") pod "maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" (UID: "36dd4024-c8de-4906-a1c6-cc0a0e757574") : secret "maas-api-e2e-mt-e16d1c0e-serving-cert" not found Nov 02 14:50:00.180557 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.180531 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dd4024-c8de-4906-a1c6-cc0a0e757574-openshift-service-ca\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.182313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.182291 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-metrics-certs\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.222288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.222225 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwcl\" (UniqueName: \"kubernetes.io/projected/36dd4024-c8de-4906-a1c6-cc0a0e757574-kube-api-access-jgwcl\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.424744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.424704 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:00.700997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.700954 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.703426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.703403 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls\") pod \"maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:00.718022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.717975 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:50:00.731014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.730956 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:50:00.891285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:00.891247 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:01.075661 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:50:01.075623 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:50:01.243145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:01.243110 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x"] Nov 02 14:50:01.243382 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:50:01.243308 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dd4024_c8de_4906_a1c6_cc0a0e757574.slice/crio-67e0a7622ada8366044cff42a1fd17cb41aa5c1402ffddd41c3012379b9f238a WatchSource:0}: Error finding container 67e0a7622ada8366044cff42a1fd17cb41aa5c1402ffddd41c3012379b9f238a: Status 404 returned error can't find the container with id 67e0a7622ada8366044cff42a1fd17cb41aa5c1402ffddd41c3012379b9f238a Nov 02 14:50:01.245010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:01.244972 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:50:01.498394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:01.498309 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" event={"ID":"36dd4024-c8de-4906-a1c6-cc0a0e757574","Type":"ContainerStarted","Data":"67e0a7622ada8366044cff42a1fd17cb41aa5c1402ffddd41c3012379b9f238a"} Nov 02 14:50:01.510136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:01.510105 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:50:02.408020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:02.407940 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:02.507446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:02.507406 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" event={"ID":"36dd4024-c8de-4906-a1c6-cc0a0e757574","Type":"ContainerStarted","Data":"654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3"} Nov 02 14:50:02.507640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:02.507550 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:02.578740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:02.578685 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" podStartSLOduration=3.05193186 podStartE2EDuration="3.578669122s" totalImagesPullingTime="526.737262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:50:02.577188888 +0000 UTC m=+3951.302810658" watchObservedRunningTime="2026-11-02 14:50:02.578669122 +0000 UTC m=+3951.304290891" Nov 02 14:50:04.323408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:04.323363 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:06.089550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:06.089507 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:07.742828 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:07.741841 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:08.519548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:08.519513 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:50:09.488569 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:09.488528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:11.605009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:11.603625 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:13.442003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:13.440527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:14.926576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:14.926540 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:15.759625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:15.759584 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:16.025004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:16.024901 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:16.205079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:16.205049 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:16.797935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:16.797893 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:17.141688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:17.141654 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt"] Nov 02 14:50:17.142226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:17.141872 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" containerID="cri-o://01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5" gracePeriod=30 Nov 02 14:50:17.211701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:17.211663 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:17.592743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:17.592595 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:17.893369 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:17.893334 5591 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:50:17.893550 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:17.893393 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:50:18.200826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:18.200723 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:18.371284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:18.371231 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:18.558598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:18.558477 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:19.445967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:19.443675 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:21.102661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:21.098880 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:22.318741 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.318708 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:22.429130 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.426284 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:50:22.533859 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.533812 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-istio-token\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-istio-token\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534040 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/874dede1-2042-4f22-8415-4e9497e0b26b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dede1-2042-4f22-8415-4e9497e0b26b-istio-podinfo\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534131 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-credential-socket\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534209 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-envoy\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534297 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-socket\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534332 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874dede1-2042-4f22-8415-4e9497e0b26b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dede1-2042-4f22-8415-4e9497e0b26b-istiod-ca-cert\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534380 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-data\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534415 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-certs\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534450 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-kube-api-access-h4fd4\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-kube-api-access-h4fd4\") pod \"874dede1-2042-4f22-8415-4e9497e0b26b\" (UID: \"874dede1-2042-4f22-8415-4e9497e0b26b\") " Nov 02 14:50:22.534913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534887 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-socket" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:50:22.535113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.534901 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-data" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:50:22.535229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.535205 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874dede1-2042-4f22-8415-4e9497e0b26b-istiod-ca-cert" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:50:22.535301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.535226 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-credential-socket" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:50:22.535471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.535448 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-certs" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:50:22.540832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.540800 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.541022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.541006 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.541146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.541132 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874dede1-2042-4f22-8415-4e9497e0b26b-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.541260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.541225 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.541362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.541349 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.541449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.541143 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-envoy" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:50:22.541544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.541246 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-kube-api-access-h4fd4" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "kube-api-access-h4fd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:50:22.541841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.541814 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-istio-token" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:50:22.542507 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.542473 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/874dede1-2042-4f22-8415-4e9497e0b26b-istio-podinfo" pod "874dede1-2042-4f22-8415-4e9497e0b26b" (UID: "874dede1-2042-4f22-8415-4e9497e0b26b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:50:22.642444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.642405 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874dede1-2042-4f22-8415-4e9497e0b26b-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.642444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.642442 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874dede1-2042-4f22-8415-4e9497e0b26b-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.642671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.642461 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4fd4\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-kube-api-access-h4fd4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.642671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.642478 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874dede1-2042-4f22-8415-4e9497e0b26b-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:50:22.713936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.713841 5591 generic.go:356] "Generic (PLEG): container finished" podID="874dede1-2042-4f22-8415-4e9497e0b26b" containerID="01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5" exitCode=0 Nov 02 14:50:22.713936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.713896 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" Nov 02 14:50:22.714187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.713924 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" event={"ID":"874dede1-2042-4f22-8415-4e9497e0b26b","Type":"ContainerDied","Data":"01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5"} Nov 02 14:50:22.714187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.714017 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt" event={"ID":"874dede1-2042-4f22-8415-4e9497e0b26b","Type":"ContainerDied","Data":"31bbe0e1d499eb8a7877e64e3d0fea13b197e6d306a29e2fff086fbe706580c6"} Nov 02 14:50:22.714187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.714046 5591 scope.go:122] "RemoveContainer" containerID="01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5" Nov 02 14:50:22.729071 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.729032 5591 scope.go:122] "RemoveContainer" containerID="366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d" Nov 02 14:50:22.739321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.739299 5591 scope.go:122] "RemoveContainer" containerID="01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5" Nov 02 14:50:22.739614 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:50:22.739592 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5\": container with ID starting with 01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5 not found: ID does not exist" containerID="01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5" Nov 02 14:50:22.739679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.739621 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5"} err="failed to get container status \"01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5\": rpc error: code = NotFound desc = could not find container \"01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5\": container with ID starting with 01e2c8450b45d4961834a176f025273d916f75cfeb062467a5701ae38ae524c5 not found: ID does not exist" Nov 02 14:50:22.739679 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.739641 5591 scope.go:122] "RemoveContainer" containerID="366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d" Nov 02 14:50:22.739893 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:50:22.739878 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d\": container with ID starting with 366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d not found: ID does not exist" containerID="366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d" Nov 02 14:50:22.739937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.739896 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d"} err="failed to get container status \"366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d\": rpc error: code = NotFound desc = could not find container \"366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d\": container with ID starting with 366ae62244b255170b87c5637c0baf22a46fb5cee073c236b2a9ad71062ca80d not found: ID does not exist" Nov 02 14:50:22.771465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.771431 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt"] Nov 02 14:50:22.784168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.784106 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-trklt"] Nov 02 14:50:22.955009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:22.954944 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:23.431996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:23.431953 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" path="/var/lib/kubelet/pods/874dede1-2042-4f22-8415-4e9497e0b26b/volumes" Nov 02 14:50:24.675742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:24.675658 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:26.157091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:26.157048 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:26.371328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:26.371197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:26.624174 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:26.624088 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:26.790221 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:26.790190 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:27.918725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:27.918691 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:28.145029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:28.144900 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:28.310655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:28.310575 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:29.887707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:29.884218 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:31.074158 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:50:31.074118 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:50:31.555535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:31.555491 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:31.624865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:31.624824 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:50:33.362712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:33.362674 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:34.506516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:34.506482 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:34.675225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:34.675190 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:35.283045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:35.283001 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:35.588382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:35.588291 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:37.388342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:37.388304 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:38.934543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:38.934505 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:40.676061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:40.675324 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:41.320573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:41.320523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:42.349875 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:42.349825 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:44.004386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:44.004351 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:45.695588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:45.694954 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:47.369037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:47.368977 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:48.993746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:48.993707 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:50.623008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:50.622956 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:52.363770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:52.363731 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:52.925457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:52.924962 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:53.770574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:53.770525 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:54.019090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:54.019054 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:54.190290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:54.190251 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:54.384810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:54.384763 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:55.701682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:55.701617 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:57.267661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:57.267623 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:50:58.867010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:50:58.852822 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:00.593842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:00.593734 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:00.821646 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:00.821597 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:01.074048 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:51:01.074012 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:51:02.178242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:02.178206 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:03.893318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:03.893285 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:05.470170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:05.470129 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:07.147009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:07.146625 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:08.771635 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:08.771598 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:10.474950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:10.474898 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:10.881423 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:10.881396 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:51:12.145608 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:12.145567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:13.282681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:13.282649 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:13.853262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:13.853160 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:15.218319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:15.218281 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:15.946082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:15.946041 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:16.677008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:16.676940 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:16.844239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:16.844206 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:17.424713 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:17.424669 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:17.702344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:17.702254 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms"] Nov 02 14:51:17.702836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:17.702534 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" containerID="cri-o://9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b" gracePeriod=30 Nov 02 14:51:17.771536 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:17.771502 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:17.937360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:17.937325 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:18.554131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:18.554095 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:19.264666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:19.264634 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:19.505065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:19.505028 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:19.688211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:19.688177 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:21.151341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:21.151140 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:21.557352 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:21.557260 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:22.812761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:22.812593 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:22.985786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:22.985714 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:51:23.068329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068288 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-certs\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068349 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-data\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068379 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-credential-socket\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068451 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8845a851-daf1-4d64-9341-ea6b4d256ce5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8845a851-daf1-4d64-9341-ea6b4d256ce5-istiod-ca-cert\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068483 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-envoy\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068525 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-kube-api-access-vb25b\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068591 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-certs" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:51:23.068845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068613 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-podinfo\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068652 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-socket\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068686 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-token\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-token\") pod \"8845a851-daf1-4d64-9341-ea6b4d256ce5\" (UID: \"8845a851-daf1-4d64-9341-ea6b4d256ce5\") " Nov 02 14:51:23.068845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068692 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-data" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:51:23.068845 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068807 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-credential-socket" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:51:23.069212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068975 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-socket" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:51:23.069212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.068897 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8845a851-daf1-4d64-9341-ea6b4d256ce5-istiod-ca-cert" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:51:23.069304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.069214 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.069304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.069235 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.069304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.069248 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.069304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.069263 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8845a851-daf1-4d64-9341-ea6b4d256ce5-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.069304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.069276 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.071404 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.071360 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-podinfo" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:51:23.071529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.071440 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-token" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:51:23.071600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.071533 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-envoy" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:51:23.071661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.071615 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-kube-api-access-vb25b" pod "8845a851-daf1-4d64-9341-ea6b4d256ce5" (UID: "8845a851-daf1-4d64-9341-ea6b4d256ce5"). InnerVolumeSpecName "kube-api-access-vb25b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:51:23.134048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.134013 5591 generic.go:356] "Generic (PLEG): container finished" podID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerID="9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b" exitCode=0 Nov 02 14:51:23.134247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.134073 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" Nov 02 14:51:23.134247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.134096 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" event={"ID":"8845a851-daf1-4d64-9341-ea6b4d256ce5","Type":"ContainerDied","Data":"9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b"} Nov 02 14:51:23.134247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.134162 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms" event={"ID":"8845a851-daf1-4d64-9341-ea6b4d256ce5","Type":"ContainerDied","Data":"6a8a3c5aec8e30bdf45ef01ccde4682f9eac66217bd04ba2f9c96525a274b011"} Nov 02 14:51:23.134247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.134195 5591 scope.go:122] "RemoveContainer" containerID="9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b" Nov 02 14:51:23.145708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.145686 5591 scope.go:122] "RemoveContainer" containerID="6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca" Nov 02 14:51:23.155538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.155513 5591 scope.go:122] "RemoveContainer" containerID="9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b" Nov 02 14:51:23.155969 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:51:23.155941 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b\": container with ID starting with 9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b not found: ID does not exist" containerID="9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b" Nov 02 14:51:23.156081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.155994 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b"} err="failed to get container status \"9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b\": rpc error: code = NotFound desc = could not find container \"9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b\": container with ID starting with 9506839cd0b43a2fc5127570df5ec6d0a2b309edafefd0fcddc37b31cd53ea9b not found: ID does not exist" Nov 02 14:51:23.156081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.156021 5591 scope.go:122] "RemoveContainer" containerID="6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca" Nov 02 14:51:23.156298 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:51:23.156271 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca\": container with ID starting with 6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca not found: ID does not exist" containerID="6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca" Nov 02 14:51:23.156343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.156306 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca"} err="failed to get container status \"6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca\": rpc error: code = NotFound desc = could not find container \"6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca\": container with ID starting with 6ba359d060222e6f8a47654f58b1fe9751473679034247d470abedb96ffc31ca not found: ID does not exist" Nov 02 14:51:23.170454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.170424 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.170454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.170451 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.170454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.170460 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8845a851-daf1-4d64-9341-ea6b4d256ce5-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.170717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.170468 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/8845a851-daf1-4d64-9341-ea6b4d256ce5-kube-api-access-vb25b\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:23.189803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.189770 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms"] Nov 02 14:51:23.195596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.195565 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-87efd3-openshift-default-6cd59997fc-9bnms"] Nov 02 14:51:23.433748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:23.433710 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" path="/var/lib/kubelet/pods/8845a851-daf1-4d64-9341-ea6b4d256ce5/volumes" Nov 02 14:51:24.565778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:24.565738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:24.788911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:24.788875 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd"] Nov 02 14:51:24.974037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:24.973975 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:25.145114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:25.145080 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:26.026554 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:26.026514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:26.282016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:26.281878 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:27.959050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:27.958976 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:28.125151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:28.125119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:28.342904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:28.342819 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:28.479443 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:28.479407 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2"] Nov 02 14:51:28.639977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:28.639947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:28.803360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:28.803323 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:29.678012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:29.674293 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:29.790217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:29.790168 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" podUID="cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" containerName="payload-processing" containerID="cri-o://84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4" gracePeriod=25 Nov 02 14:51:31.074581 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:51:31.074543 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:51:31.366062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:31.356804 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:31.968886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:31.968850 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:32.945354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:32.945313 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:33.322777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:33.322687 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" podUID="cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:51:33.480359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:33.480285 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" podUID="7a879895-0675-4c59-a0f4-ab94b9846d47" containerName="payload-pre-processing" containerID="cri-o://77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc" gracePeriod=25 Nov 02 14:51:34.591539 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:34.591302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:35.027604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:35.027566 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:36.422080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:36.420768 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:38.196237 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:38.196163 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:39.773793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:39.773756 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:41.405056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:41.405015 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:42.896316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:42.896272 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" podUID="7a879895-0675-4c59-a0f4-ab94b9846d47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:51:43.031612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:43.031542 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:43.322486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:43.322386 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" podUID="cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:51:44.659217 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:44.659173 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:46.271010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:46.261469 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:47.967086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:47.967046 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:49.651006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:49.646175 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:50.068866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.068836 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:51:50.193660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.193567 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-plugins-config-volume\") pod \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\" (UID: \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\") " Nov 02 14:51:50.194464 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.194437 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-kube-api-access-x7hb9\" (UniqueName: \"kubernetes.io/projected/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-kube-api-access-x7hb9\") pod \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\" (UID: \"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77\") " Nov 02 14:51:50.195049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.194265 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-plugins-config-volume" pod "cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" (UID: "cdc2c0ac-28c8-4708-b02e-eb77d97fbe77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:51:50.197078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.197054 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-kube-api-access-x7hb9" pod "cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" (UID: "cdc2c0ac-28c8-4708-b02e-eb77d97fbe77"). InnerVolumeSpecName "kube-api-access-x7hb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:51:50.295818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.295778 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:50.295818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.295813 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hb9\" (UniqueName: \"kubernetes.io/projected/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77-kube-api-access-x7hb9\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:50.327933 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.327888 5591 generic.go:356] "Generic (PLEG): container finished" podID="cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" containerID="84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4" exitCode=0 Nov 02 14:51:50.328176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.327946 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" Nov 02 14:51:50.328176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.327976 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" event={"ID":"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77","Type":"ContainerDied","Data":"84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4"} Nov 02 14:51:50.328176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.328072 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd" event={"ID":"cdc2c0ac-28c8-4708-b02e-eb77d97fbe77","Type":"ContainerDied","Data":"8e61f567b045450fe7fdab7382fc3c9b62402656afd1d3f45f2a08ca0ab3f6d2"} Nov 02 14:51:50.328176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.328103 5591 scope.go:122] "RemoveContainer" containerID="84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4" Nov 02 14:51:50.338950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.338928 5591 scope.go:122] "RemoveContainer" containerID="84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4" Nov 02 14:51:50.339267 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:51:50.339246 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4\": container with ID starting with 84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4 not found: ID does not exist" containerID="84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4" Nov 02 14:51:50.339386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.339281 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4"} err="failed to get container status \"84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4\": rpc error: code = NotFound desc = could not find container \"84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4\": container with ID starting with 84eb2d52a57cca0611d5fd776b90beb17ddfe7fad6fb60e9868a52ad332bbcf4 not found: ID does not exist" Nov 02 14:51:50.382687 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.382648 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd"] Nov 02 14:51:50.389103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:50.389072 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bc34db-598db67bb9-84wtd"] Nov 02 14:51:51.203248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:51.203204 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:51.432095 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:51.432061 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" path="/var/lib/kubelet/pods/cdc2c0ac-28c8-4708-b02e-eb77d97fbe77/volumes" Nov 02 14:51:52.820793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:52.820530 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:52.896819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:52.896779 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" podUID="7a879895-0675-4c59-a0f4-ab94b9846d47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:51:53.769761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:53.769732 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:51:53.843522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:53.843487 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a879895-0675-4c59-a0f4-ab94b9846d47-kube-api-access-8m2d7\" (UniqueName: \"kubernetes.io/projected/7a879895-0675-4c59-a0f4-ab94b9846d47-kube-api-access-8m2d7\") pod \"7a879895-0675-4c59-a0f4-ab94b9846d47\" (UID: \"7a879895-0675-4c59-a0f4-ab94b9846d47\") " Nov 02 14:51:53.844062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:53.843616 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a879895-0675-4c59-a0f4-ab94b9846d47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a879895-0675-4c59-a0f4-ab94b9846d47-plugins-config-volume\") pod \"7a879895-0675-4c59-a0f4-ab94b9846d47\" (UID: \"7a879895-0675-4c59-a0f4-ab94b9846d47\") " Nov 02 14:51:53.844062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:53.844014 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a879895-0675-4c59-a0f4-ab94b9846d47-plugins-config-volume" pod "7a879895-0675-4c59-a0f4-ab94b9846d47" (UID: "7a879895-0675-4c59-a0f4-ab94b9846d47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:51:53.844235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:53.844219 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a879895-0675-4c59-a0f4-ab94b9846d47-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:53.846327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:53.846297 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a879895-0675-4c59-a0f4-ab94b9846d47-kube-api-access-8m2d7" pod "7a879895-0675-4c59-a0f4-ab94b9846d47" (UID: "7a879895-0675-4c59-a0f4-ab94b9846d47"). InnerVolumeSpecName "kube-api-access-8m2d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:51:53.945283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:53.945245 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m2d7\" (UniqueName: \"kubernetes.io/projected/7a879895-0675-4c59-a0f4-ab94b9846d47-kube-api-access-8m2d7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:51:54.362468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.362430 5591 generic.go:356] "Generic (PLEG): container finished" podID="7a879895-0675-4c59-a0f4-ab94b9846d47" containerID="77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc" exitCode=0 Nov 02 14:51:54.362681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.362477 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" Nov 02 14:51:54.362681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.362515 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" event={"ID":"7a879895-0675-4c59-a0f4-ab94b9846d47","Type":"ContainerDied","Data":"77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc"} Nov 02 14:51:54.362681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.362574 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2" event={"ID":"7a879895-0675-4c59-a0f4-ab94b9846d47","Type":"ContainerDied","Data":"2b51b5a2ef008ea6d207de9b4e15eca197d01e43c991960fc75e9f676d13fb0f"} Nov 02 14:51:54.362681 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.362595 5591 scope.go:122] "RemoveContainer" containerID="77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc" Nov 02 14:51:54.373157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.373131 5591 scope.go:122] "RemoveContainer" containerID="77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc" Nov 02 14:51:54.373442 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:51:54.373421 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc\": container with ID starting with 77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc not found: ID does not exist" containerID="77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc" Nov 02 14:51:54.373483 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.373452 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc"} err="failed to get container status \"77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc\": rpc error: code = NotFound desc = could not find container \"77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc\": container with ID starting with 77a589172eeccdca906b63ec6046e2ca356033748263bdb67685a74bb79c0bbc not found: ID does not exist" Nov 02 14:51:54.417254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.417216 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2"] Nov 02 14:51:54.430135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.430103 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d13f4-6d879f5b87-sd2x2"] Nov 02 14:51:54.471893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:54.471811 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:55.433617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:55.433575 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a879895-0675-4c59-a0f4-ab94b9846d47" path="/var/lib/kubelet/pods/7a879895-0675-4c59-a0f4-ab94b9846d47/volumes" Nov 02 14:51:56.153680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:56.153333 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:57.751270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:57.751230 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:51:59.474589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:51:59.474557 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:01.066044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:01.063997 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:01.116718 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:52:01.116673 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:52:02.671661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:02.671591 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:04.331090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:04.331052 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:04.496528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:04.496487 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:52:05.770453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:05.770411 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:06.224043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:06.222168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:06.770184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:06.770143 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:08.379478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:08.379408 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:08.996031 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:08.995966 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:09.158671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:09.158637 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:09.768442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:09.768396 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:10.053898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:10.053794 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:10.270477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:10.270436 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:10.957313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:10.957150 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:11.878611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:11.878562 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:13.446534 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:13.446492 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:14.955444 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:14.955405 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:15.574305 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:15.574268 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:15.755849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:15.755802 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:16.387351 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:16.387298 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:16.746524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:16.746431 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:16.924993 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:16.924936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:17.451042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:17.451007 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b"] Nov 02 14:52:17.573597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:17.573557 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:17.732381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:17.732299 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:17.934189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:17.934158 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:18.592977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:18.592938 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:20.288033 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:20.274548 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:21.878274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:21.878191 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:22.451886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.451847 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" podUID="d33bbdce-378e-4067-8323-3a1a9892f2e6" containerName="payload-pre-processing" containerID="cri-o://d1b482f6cab27d6da149f6cd8cd20272a95877a3d5883331affa8a7e04484c0f" gracePeriod=25 Nov 02 14:52:22.677089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.676974 5591 generic.go:356] "Generic (PLEG): container finished" podID="d33bbdce-378e-4067-8323-3a1a9892f2e6" containerID="d1b482f6cab27d6da149f6cd8cd20272a95877a3d5883331affa8a7e04484c0f" exitCode=0 Nov 02 14:52:22.677089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.677097 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" event={"ID":"d33bbdce-378e-4067-8323-3a1a9892f2e6","Type":"ContainerDied","Data":"d1b482f6cab27d6da149f6cd8cd20272a95877a3d5883331affa8a7e04484c0f"} Nov 02 14:52:22.733231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.733203 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:52:22.831245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.831210 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33bbdce-378e-4067-8323-3a1a9892f2e6-kube-api-access-vss49\" (UniqueName: \"kubernetes.io/projected/d33bbdce-378e-4067-8323-3a1a9892f2e6-kube-api-access-vss49\") pod \"d33bbdce-378e-4067-8323-3a1a9892f2e6\" (UID: \"d33bbdce-378e-4067-8323-3a1a9892f2e6\") " Nov 02 14:52:22.831467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.831396 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33bbdce-378e-4067-8323-3a1a9892f2e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33bbdce-378e-4067-8323-3a1a9892f2e6-plugins-config-volume\") pod \"d33bbdce-378e-4067-8323-3a1a9892f2e6\" (UID: \"d33bbdce-378e-4067-8323-3a1a9892f2e6\") " Nov 02 14:52:22.831836 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.831811 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33bbdce-378e-4067-8323-3a1a9892f2e6-plugins-config-volume" pod "d33bbdce-378e-4067-8323-3a1a9892f2e6" (UID: "d33bbdce-378e-4067-8323-3a1a9892f2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:52:22.831972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.831958 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33bbdce-378e-4067-8323-3a1a9892f2e6-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:22.833418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.833391 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33bbdce-378e-4067-8323-3a1a9892f2e6-kube-api-access-vss49" pod "d33bbdce-378e-4067-8323-3a1a9892f2e6" (UID: "d33bbdce-378e-4067-8323-3a1a9892f2e6"). InnerVolumeSpecName "kube-api-access-vss49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:52:22.933641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:22.933605 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vss49\" (UniqueName: \"kubernetes.io/projected/d33bbdce-378e-4067-8323-3a1a9892f2e6-kube-api-access-vss49\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:23.316819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:23.316780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:23.728202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:23.728159 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" event={"ID":"d33bbdce-378e-4067-8323-3a1a9892f2e6","Type":"ContainerDied","Data":"b27906c8a61e6b33387a403ece50d68fb6e2c5f7c0a835808ea3c5971ef07595"} Nov 02 14:52:23.728494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:23.728466 5591 scope.go:122] "RemoveContainer" containerID="d1b482f6cab27d6da149f6cd8cd20272a95877a3d5883331affa8a7e04484c0f" Nov 02 14:52:23.728689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:23.728670 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b" Nov 02 14:52:23.794960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:23.794508 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b"] Nov 02 14:52:23.805261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:23.805224 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-605541-59c4dbc5b7-s8d6b"] Nov 02 14:52:24.021906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:24.021812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:24.190978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:24.190942 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:24.885438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:24.885405 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:25.082590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:25.082560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:25.290804 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:25.290722 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:25.436293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:25.436233 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33bbdce-378e-4067-8323-3a1a9892f2e6" path="/var/lib/kubelet/pods/d33bbdce-378e-4067-8323-3a1a9892f2e6/volumes" Nov 02 14:52:25.839751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:25.839715 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:27.582314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:27.582277 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:28.146562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:28.146525 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:28.520879 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:28.520798 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:29.234257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:29.234222 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:30.808333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:30.808296 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:31.075916 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:52:31.075829 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:52:31.328900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:31.328803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:31.845639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:31.845599 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82"] Nov 02 14:52:31.874997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:31.874952 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j"] Nov 02 14:52:32.033580 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:32.033549 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:32.205858 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:32.205822 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:32.688361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:32.688322 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:32.862194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:32.862159 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:33.497527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:33.497491 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:33.814591 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:33.814510 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:34.610784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:34.610749 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:34.793298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:34.793264 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:36.142458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:36.142410 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:36.845885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:36.845834 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" podUID="9d6b9d59-c7f6-4f43-b2b7-85137885ec3a" containerName="payload-processing" containerID="cri-o://de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055" gracePeriod=25 Nov 02 14:52:36.875710 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:36.875674 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" podUID="17d10963-ca4b-4b84-ae31-a8b9981f78b3" containerName="payload-pre-processing" containerID="cri-o://d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179" gracePeriod=25 Nov 02 14:52:37.155931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.155903 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:52:37.185833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.185806 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:52:37.303593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.303556 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-kube-api-access-l2lz5\" (UniqueName: \"kubernetes.io/projected/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-kube-api-access-l2lz5\") pod \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\" (UID: \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\") " Nov 02 14:52:37.303795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.303640 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d10963-ca4b-4b84-ae31-a8b9981f78b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d10963-ca4b-4b84-ae31-a8b9981f78b3-plugins-config-volume\") pod \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\" (UID: \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\") " Nov 02 14:52:37.303795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.303763 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d10963-ca4b-4b84-ae31-a8b9981f78b3-kube-api-access-wzbq9\" (UniqueName: \"kubernetes.io/projected/17d10963-ca4b-4b84-ae31-a8b9981f78b3-kube-api-access-wzbq9\") pod \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\" (UID: \"17d10963-ca4b-4b84-ae31-a8b9981f78b3\") " Nov 02 14:52:37.303932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.303810 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-plugins-config-volume\") pod \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\" (UID: \"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a\") " Nov 02 14:52:37.304096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.304060 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d10963-ca4b-4b84-ae31-a8b9981f78b3-plugins-config-volume" pod "17d10963-ca4b-4b84-ae31-a8b9981f78b3" (UID: "17d10963-ca4b-4b84-ae31-a8b9981f78b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:52:37.304313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.304289 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-plugins-config-volume" pod "9d6b9d59-c7f6-4f43-b2b7-85137885ec3a" (UID: "9d6b9d59-c7f6-4f43-b2b7-85137885ec3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:52:37.304449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.304433 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d10963-ca4b-4b84-ae31-a8b9981f78b3-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:37.304525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.304451 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:37.305899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.305878 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-kube-api-access-l2lz5" pod "9d6b9d59-c7f6-4f43-b2b7-85137885ec3a" (UID: "9d6b9d59-c7f6-4f43-b2b7-85137885ec3a"). InnerVolumeSpecName "kube-api-access-l2lz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:52:37.305978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.305906 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d10963-ca4b-4b84-ae31-a8b9981f78b3-kube-api-access-wzbq9" pod "17d10963-ca4b-4b84-ae31-a8b9981f78b3" (UID: "17d10963-ca4b-4b84-ae31-a8b9981f78b3"). InnerVolumeSpecName "kube-api-access-wzbq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:52:37.405319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.405285 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2lz5\" (UniqueName: \"kubernetes.io/projected/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a-kube-api-access-l2lz5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:37.405319 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.405315 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbq9\" (UniqueName: \"kubernetes.io/projected/17d10963-ca4b-4b84-ae31-a8b9981f78b3-kube-api-access-wzbq9\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:37.846868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.846726 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:37.908598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.908562 5591 generic.go:356] "Generic (PLEG): container finished" podID="17d10963-ca4b-4b84-ae31-a8b9981f78b3" containerID="d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179" exitCode=0 Nov 02 14:52:37.908780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.908618 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" Nov 02 14:52:37.908780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.908648 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" event={"ID":"17d10963-ca4b-4b84-ae31-a8b9981f78b3","Type":"ContainerDied","Data":"d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179"} Nov 02 14:52:37.908780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.908708 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j" event={"ID":"17d10963-ca4b-4b84-ae31-a8b9981f78b3","Type":"ContainerDied","Data":"ade3e36ca208b941547980f28959bf199f4ceeb353e169319087ddfe19ca88af"} Nov 02 14:52:37.908780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.908738 5591 scope.go:122] "RemoveContainer" containerID="d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179" Nov 02 14:52:37.910524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.910493 5591 generic.go:356] "Generic (PLEG): container finished" podID="9d6b9d59-c7f6-4f43-b2b7-85137885ec3a" containerID="de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055" exitCode=0 Nov 02 14:52:37.910670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.910534 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" Nov 02 14:52:37.910670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.910530 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" event={"ID":"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a","Type":"ContainerDied","Data":"de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055"} Nov 02 14:52:37.910792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.910709 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82" event={"ID":"9d6b9d59-c7f6-4f43-b2b7-85137885ec3a","Type":"ContainerDied","Data":"138a69463a5fa6ff4c649571a889e415e7ab2c678c14f7bc78b6c7896932b767"} Nov 02 14:52:37.919180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.919151 5591 scope.go:122] "RemoveContainer" containerID="d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179" Nov 02 14:52:37.919452 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:52:37.919428 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179\": container with ID starting with d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179 not found: ID does not exist" containerID="d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179" Nov 02 14:52:37.919514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.919460 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179"} err="failed to get container status \"d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179\": rpc error: code = NotFound desc = could not find container \"d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179\": container with ID starting with d2a743426cd8b72fa319b93a7e707b6410966d06d253e1c65b92e633d27ab179 not found: ID does not exist" Nov 02 14:52:37.919514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.919477 5591 scope.go:122] "RemoveContainer" containerID="de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055" Nov 02 14:52:37.929034 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.929016 5591 scope.go:122] "RemoveContainer" containerID="de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055" Nov 02 14:52:37.929303 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:52:37.929284 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055\": container with ID starting with de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055 not found: ID does not exist" containerID="de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055" Nov 02 14:52:37.929360 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.929310 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055"} err="failed to get container status \"de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055\": rpc error: code = NotFound desc = could not find container \"de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055\": container with ID starting with de93adc9372de5560074095e5df4b22948bd76b09514ebced488878315ae0055 not found: ID does not exist" Nov 02 14:52:37.962514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.962489 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82"] Nov 02 14:52:37.965202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.965176 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e16d1c0e-654b584fbf-m7g82"] Nov 02 14:52:37.997866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:37.997835 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j"] Nov 02 14:52:38.003776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:38.003752 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e16d1c0e-77896f8bc5-zkm5j"] Nov 02 14:52:38.563336 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:38.563298 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:39.470265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:39.470218 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:39.473712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:39.473675 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d10963-ca4b-4b84-ae31-a8b9981f78b3" path="/var/lib/kubelet/pods/17d10963-ca4b-4b84-ae31-a8b9981f78b3/volumes" Nov 02 14:52:39.474485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:39.474454 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6b9d59-c7f6-4f43-b2b7-85137885ec3a" path="/var/lib/kubelet/pods/9d6b9d59-c7f6-4f43-b2b7-85137885ec3a/volumes" Nov 02 14:52:40.099253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:40.099189 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:40.261627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:40.261591 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:41.169162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:41.169125 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:41.373690 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:41.373638 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:41.949207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:41.949162 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:42.979628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:42.979589 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:44.571183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:44.571149 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:46.069633 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:46.069597 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:46.254455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:46.254417 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:47.967667 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:47.967627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:48.381109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:48.381080 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:49.576009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:49.575265 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:51.272921 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:51.272857 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:52.602777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.602740 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x"] Nov 02 14:52:52.603283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.603023 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" podUID="36dd4024-c8de-4906-a1c6-cc0a0e757574" containerName="maas-api" containerID="cri-o://654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3" gracePeriod=30 Nov 02 14:52:52.720071 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.720026 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:52.956899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.956870 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:52:52.968959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.968929 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-metrics-certs\") pod \"36dd4024-c8de-4906-a1c6-cc0a0e757574\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " Nov 02 14:52:52.969132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.969014 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls\") pod \"36dd4024-c8de-4906-a1c6-cc0a0e757574\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " Nov 02 14:52:52.969132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.969105 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dd4024-c8de-4906-a1c6-cc0a0e757574-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dd4024-c8de-4906-a1c6-cc0a0e757574-openshift-service-ca\") pod \"36dd4024-c8de-4906-a1c6-cc0a0e757574\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " Nov 02 14:52:52.969293 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.969275 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dd4024-c8de-4906-a1c6-cc0a0e757574-kube-api-access-jgwcl\" (UniqueName: \"kubernetes.io/projected/36dd4024-c8de-4906-a1c6-cc0a0e757574-kube-api-access-jgwcl\") pod \"36dd4024-c8de-4906-a1c6-cc0a0e757574\" (UID: \"36dd4024-c8de-4906-a1c6-cc0a0e757574\") " Nov 02 14:52:52.969520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.969492 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dd4024-c8de-4906-a1c6-cc0a0e757574-openshift-service-ca" pod "36dd4024-c8de-4906-a1c6-cc0a0e757574" (UID: "36dd4024-c8de-4906-a1c6-cc0a0e757574"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:52:52.969846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.969823 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dd4024-c8de-4906-a1c6-cc0a0e757574-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:52.971276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.971196 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-metrics-certs" pod "36dd4024-c8de-4906-a1c6-cc0a0e757574" (UID: "36dd4024-c8de-4906-a1c6-cc0a0e757574"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:52:52.972537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.971425 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dd4024-c8de-4906-a1c6-cc0a0e757574-kube-api-access-jgwcl" pod "36dd4024-c8de-4906-a1c6-cc0a0e757574" (UID: "36dd4024-c8de-4906-a1c6-cc0a0e757574"). InnerVolumeSpecName "kube-api-access-jgwcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:52:52.972688 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:52.972665 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls" pod "36dd4024-c8de-4906-a1c6-cc0a0e757574" (UID: "36dd4024-c8de-4906-a1c6-cc0a0e757574"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:52:53.075602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.075512 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgwcl\" (UniqueName: \"kubernetes.io/projected/36dd4024-c8de-4906-a1c6-cc0a0e757574-kube-api-access-jgwcl\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:53.075602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.075551 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:53.075602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.075567 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dd4024-c8de-4906-a1c6-cc0a0e757574-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:52:53.110023 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.109947 5591 generic.go:356] "Generic (PLEG): container finished" podID="36dd4024-c8de-4906-a1c6-cc0a0e757574" containerID="654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3" exitCode=0 Nov 02 14:52:53.121403 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.110435 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" Nov 02 14:52:53.156165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.110452 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" event={"ID":"36dd4024-c8de-4906-a1c6-cc0a0e757574","Type":"ContainerDied","Data":"654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3"} Nov 02 14:52:53.156165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.131419 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x" event={"ID":"36dd4024-c8de-4906-a1c6-cc0a0e757574","Type":"ContainerDied","Data":"67e0a7622ada8366044cff42a1fd17cb41aa5c1402ffddd41c3012379b9f238a"} Nov 02 14:52:53.156165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.131466 5591 scope.go:122] "RemoveContainer" containerID="654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3" Nov 02 14:52:53.181564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.181267 5591 scope.go:122] "RemoveContainer" containerID="654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3" Nov 02 14:52:53.181720 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:52:53.181684 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3\": container with ID starting with 654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3 not found: ID does not exist" containerID="654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3" Nov 02 14:52:53.181775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.181737 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3"} err="failed to get container status \"654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3\": rpc error: code = NotFound desc = could not find container \"654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3\": container with ID starting with 654dbddfd15afc277dc8aef4c6e0e5efd1b24e3e1aa59383b7e425fb5e3413e3 not found: ID does not exist" Nov 02 14:52:53.214904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.214868 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x"] Nov 02 14:52:53.218698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.218671 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e16d1c0e-64457fb7b5-5md4x"] Nov 02 14:52:53.432056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.426346 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:53.449442 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:53.449404 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36dd4024-c8de-4906-a1c6-cc0a0e757574" path="/var/lib/kubelet/pods/36dd4024-c8de-4906-a1c6-cc0a0e757574/volumes" Nov 02 14:52:54.646010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:54.645953 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:52:54.840175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:54.840143 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:55.006468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:55.006381 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:55.554376 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:55.554341 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:55.939745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:55.939698 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:57.161135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:57.161097 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:52:58.853931 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:52:58.853882 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:00.129003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.128951 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f"] Nov 02 14:53:00.129519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.129245 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" podUID="95c695c6-5acc-481e-a4b0-06c5fe2d933b" containerName="maas-api" containerID="cri-o://e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e" gracePeriod=30 Nov 02 14:53:00.264835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.264803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:00.468513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.468479 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:53:00.558062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.558007 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c695c6-5acc-481e-a4b0-06c5fe2d933b-kube-api-access-tpzz5\" (UniqueName: \"kubernetes.io/projected/95c695c6-5acc-481e-a4b0-06c5fe2d933b-kube-api-access-tpzz5\") pod \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " Nov 02 14:53:00.558479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.558450 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls\") pod \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " Nov 02 14:53:00.558735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.558719 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-metrics-certs\") pod \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " Nov 02 14:53:00.558940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.558923 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c695c6-5acc-481e-a4b0-06c5fe2d933b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c695c6-5acc-481e-a4b0-06c5fe2d933b-openshift-service-ca\") pod \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\" (UID: \"95c695c6-5acc-481e-a4b0-06c5fe2d933b\") " Nov 02 14:53:00.563543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.563365 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c695c6-5acc-481e-a4b0-06c5fe2d933b-kube-api-access-tpzz5" pod "95c695c6-5acc-481e-a4b0-06c5fe2d933b" (UID: "95c695c6-5acc-481e-a4b0-06c5fe2d933b"). InnerVolumeSpecName "kube-api-access-tpzz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:00.564013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.563688 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c695c6-5acc-481e-a4b0-06c5fe2d933b-openshift-service-ca" pod "95c695c6-5acc-481e-a4b0-06c5fe2d933b" (UID: "95c695c6-5acc-481e-a4b0-06c5fe2d933b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:53:00.663006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.662882 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c695c6-5acc-481e-a4b0-06c5fe2d933b-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:00.663006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.662926 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpzz5\" (UniqueName: \"kubernetes.io/projected/95c695c6-5acc-481e-a4b0-06c5fe2d933b-kube-api-access-tpzz5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:00.681394 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.681209 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-metrics-certs" pod "95c695c6-5acc-481e-a4b0-06c5fe2d933b" (UID: "95c695c6-5acc-481e-a4b0-06c5fe2d933b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:53:00.681920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.681889 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls" pod "95c695c6-5acc-481e-a4b0-06c5fe2d933b" (UID: "95c695c6-5acc-481e-a4b0-06c5fe2d933b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:53:00.698574 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.698484 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:00.764619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.764520 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:00.764619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:00.764556 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c695c6-5acc-481e-a4b0-06c5fe2d933b-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:01.104898 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:53:01.104806 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:53:01.180632 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.180595 5591 generic.go:356] "Generic (PLEG): container finished" podID="95c695c6-5acc-481e-a4b0-06c5fe2d933b" containerID="e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e" exitCode=0 Nov 02 14:53:01.181137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.180638 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" Nov 02 14:53:01.181137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.180678 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" event={"ID":"95c695c6-5acc-481e-a4b0-06c5fe2d933b","Type":"ContainerDied","Data":"e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e"} Nov 02 14:53:01.181137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.180732 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f" event={"ID":"95c695c6-5acc-481e-a4b0-06c5fe2d933b","Type":"ContainerDied","Data":"e06ed40d9ea082628c1deda8c97d10f669e41a8137b240c12d05c656d973a8a9"} Nov 02 14:53:01.181137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.180761 5591 scope.go:122] "RemoveContainer" containerID="e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e" Nov 02 14:53:01.191848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.191826 5591 scope.go:122] "RemoveContainer" containerID="e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e" Nov 02 14:53:01.192153 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:53:01.192131 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e\": container with ID starting with e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e not found: ID does not exist" containerID="e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e" Nov 02 14:53:01.192225 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.192164 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e"} err="failed to get container status \"e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e\": rpc error: code = NotFound desc = could not find container \"e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e\": container with ID starting with e5b3e454a4d1e3ff21e24b70594b584447aa6c939ad5a40e882c3bceef72c41e not found: ID does not exist" Nov 02 14:53:01.233001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.232948 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f"] Nov 02 14:53:01.245052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.245019 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a4809a-7854fd9b56-j768f"] Nov 02 14:53:01.342137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.342103 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:01.435871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:01.435837 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c695c6-5acc-481e-a4b0-06c5fe2d933b" path="/var/lib/kubelet/pods/95c695c6-5acc-481e-a4b0-06c5fe2d933b/volumes" Nov 02 14:53:03.073968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:03.065328 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:04.553669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:04.552485 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:04.724834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:04.724803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:04.893664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:04.893623 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:05.311427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:05.311376 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:05.754005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:05.733339 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:06.563015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:06.560718 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:06.728265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:06.728231 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:07.808599 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:07.808553 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:07.991256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:07.991221 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:08.172572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:08.172528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:08.516740 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:08.516627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:08.810050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:08.809944 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m"] Nov 02 14:53:08.810523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:08.810208 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" containerID="cri-o://f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4" gracePeriod=30 Nov 02 14:53:08.950203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:08.947488 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:09.540175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:09.540144 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:09.721043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:09.720832 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:10.092040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:10.088958 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:11.860315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:11.860088 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:12.615868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:12.615831 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:12.801401 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:12.801367 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp"] Nov 02 14:53:12.996126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:12.996091 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:13.158163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:13.158127 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:13.419113 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:13.419077 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6"] Nov 02 14:53:13.419626 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:13.419594 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" podUID="c30329ab-6930-4243-826e-4f1733f6172a" containerName="maas-api" containerID="cri-o://f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7" gracePeriod=30 Nov 02 14:53:13.550551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:13.533202 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:13.990203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:13.990169 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:53:14.008783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.008755 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-maas-api-tls\") pod \"c30329ab-6930-4243-826e-4f1733f6172a\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " Nov 02 14:53:14.012517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.012481 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-maas-api-tls" pod "c30329ab-6930-4243-826e-4f1733f6172a" (UID: "c30329ab-6930-4243-826e-4f1733f6172a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:53:14.110262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.110227 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-metrics-certs\") pod \"c30329ab-6930-4243-826e-4f1733f6172a\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " Nov 02 14:53:14.110448 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.110435 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30329ab-6930-4243-826e-4f1733f6172a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c30329ab-6930-4243-826e-4f1733f6172a-openshift-service-ca\") pod \"c30329ab-6930-4243-826e-4f1733f6172a\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " Nov 02 14:53:14.110491 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.110477 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30329ab-6930-4243-826e-4f1733f6172a-kube-api-access-8vs56\" (UniqueName: \"kubernetes.io/projected/c30329ab-6930-4243-826e-4f1733f6172a-kube-api-access-8vs56\") pod \"c30329ab-6930-4243-826e-4f1733f6172a\" (UID: \"c30329ab-6930-4243-826e-4f1733f6172a\") " Nov 02 14:53:14.110814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.110788 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30329ab-6930-4243-826e-4f1733f6172a-openshift-service-ca" pod "c30329ab-6930-4243-826e-4f1733f6172a" (UID: "c30329ab-6930-4243-826e-4f1733f6172a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:53:14.110920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.110865 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.112955 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.112930 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-metrics-certs" pod "c30329ab-6930-4243-826e-4f1733f6172a" (UID: "c30329ab-6930-4243-826e-4f1733f6172a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:53:14.113089 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.113052 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30329ab-6930-4243-826e-4f1733f6172a-kube-api-access-8vs56" pod "c30329ab-6930-4243-826e-4f1733f6172a" (UID: "c30329ab-6930-4243-826e-4f1733f6172a"). InnerVolumeSpecName "kube-api-access-8vs56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:14.145064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.145037 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:53:14.219056 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.218923 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:14.223162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.222226 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c30329ab-6930-4243-826e-4f1733f6172a-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.223162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.222262 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vs56\" (UniqueName: \"kubernetes.io/projected/c30329ab-6930-4243-826e-4f1733f6172a-kube-api-access-8vs56\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.223162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.222278 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30329ab-6930-4243-826e-4f1733f6172a-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.267587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.267548 5591 generic.go:356] "Generic (PLEG): container finished" podID="c30329ab-6930-4243-826e-4f1733f6172a" containerID="f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7" exitCode=0 Nov 02 14:53:14.267785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.267598 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" Nov 02 14:53:14.267785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.267630 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" event={"ID":"c30329ab-6930-4243-826e-4f1733f6172a","Type":"ContainerDied","Data":"f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7"} Nov 02 14:53:14.267785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.267691 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6" event={"ID":"c30329ab-6930-4243-826e-4f1733f6172a","Type":"ContainerDied","Data":"091761f680f03ea77b828063b3afceb9aece209a4393f583eb3a3ad8415e4378"} Nov 02 14:53:14.267785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.267722 5591 scope.go:122] "RemoveContainer" containerID="f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7" Nov 02 14:53:14.269791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.269762 5591 generic.go:356] "Generic (PLEG): container finished" podID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerID="f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4" exitCode=0 Nov 02 14:53:14.269909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.269826 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" Nov 02 14:53:14.269964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.269824 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" event={"ID":"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c","Type":"ContainerDied","Data":"f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4"} Nov 02 14:53:14.270085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.269969 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m" event={"ID":"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c","Type":"ContainerDied","Data":"92a4d77ff668bb4d638ebe907f13827a14a1f2b83ea4145fe6f7e5515c3e601a"} Nov 02 14:53:14.278426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.278403 5591 scope.go:122] "RemoveContainer" containerID="f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7" Nov 02 14:53:14.278695 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:53:14.278676 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7\": container with ID starting with f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7 not found: ID does not exist" containerID="f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7" Nov 02 14:53:14.278743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.278703 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7"} err="failed to get container status \"f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7\": rpc error: code = NotFound desc = could not find container \"f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7\": container with ID starting with f19462454781776f2281150268421a055d15d3241968c477acaf41562bf11be7 not found: ID does not exist" Nov 02 14:53:14.278743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.278722 5591 scope.go:122] "RemoveContainer" containerID="f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4" Nov 02 14:53:14.288634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.288616 5591 scope.go:122] "RemoveContainer" containerID="5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7" Nov 02 14:53:14.297477 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.297454 5591 scope.go:122] "RemoveContainer" containerID="f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4" Nov 02 14:53:14.297748 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:53:14.297729 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4\": container with ID starting with f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4 not found: ID does not exist" containerID="f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4" Nov 02 14:53:14.297806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.297757 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4"} err="failed to get container status \"f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4\": rpc error: code = NotFound desc = could not find container \"f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4\": container with ID starting with f0e29241285304efc595313ee5f009363cb46caff31d9800d61442c4e73c1cc4 not found: ID does not exist" Nov 02 14:53:14.297806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.297774 5591 scope.go:122] "RemoveContainer" containerID="5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7" Nov 02 14:53:14.298084 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:53:14.298054 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7\": container with ID starting with 5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7 not found: ID does not exist" containerID="5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7" Nov 02 14:53:14.298158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.298092 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7"} err="failed to get container status \"5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7\": rpc error: code = NotFound desc = could not find container \"5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7\": container with ID starting with 5f0fb2ce3c76aca75bcc71705f25129d5ad9ee4bc21494ae1411a47ac3a524d7 not found: ID does not exist" Nov 02 14:53:14.319366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.319339 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6"] Nov 02 14:53:14.322699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.322667 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-socket\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.322838 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.322791 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-kube-api-access-8gtrv\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-kube-api-access-8gtrv\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.322903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.322835 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-certs\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.322903 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.322871 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-podinfo\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.323008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.322921 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-socket" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:14.323074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323003 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-data\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.323074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323050 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-token\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-token\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.323202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323072 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istiod-ca-cert\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.323202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323164 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-credential-socket\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.323202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323184 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-certs" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:14.323379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323205 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-envoy\") pod \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\" (UID: \"3317507b-9f40-45b7-a3dc-9bfd1fc20c9c\") " Nov 02 14:53:14.323379 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323316 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-data" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:14.323502 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323471 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-credential-socket" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:14.323558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323530 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istiod-ca-cert" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:53:14.323801 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323781 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.323897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323807 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.323897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323823 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.323897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323837 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.323897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.323851 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.325218 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.325192 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-podinfo" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:53:14.325342 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.325264 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-envoy" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:14.325451 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.325434 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-kube-api-access-8gtrv" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "kube-api-access-8gtrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:14.326191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.326169 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-662e3e-6d9f44c9cf-w78z6"] Nov 02 14:53:14.326457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.326440 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-token" pod "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" (UID: "3317507b-9f40-45b7-a3dc-9bfd1fc20c9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:14.424897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.424864 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.424897 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.424896 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.425210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.424909 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gtrv\" (UniqueName: \"kubernetes.io/projected/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-kube-api-access-8gtrv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.425210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.424922 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:14.624090 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.624062 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m"] Nov 02 14:53:14.641159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:14.637587 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4b7759-openshift-default-58886bd6f9-zxw7m"] Nov 02 14:53:15.432040 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:15.431998 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" path="/var/lib/kubelet/pods/3317507b-9f40-45b7-a3dc-9bfd1fc20c9c/volumes" Nov 02 14:53:15.432701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:15.432678 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30329ab-6930-4243-826e-4f1733f6172a" path="/var/lib/kubelet/pods/c30329ab-6930-4243-826e-4f1733f6172a/volumes" Nov 02 14:53:15.492824 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:15.492782 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p"] Nov 02 14:53:15.684216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:15.684141 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:15.849805 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:15.849776 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:16.094059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:16.092976 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:16.457471 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:16.457433 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:17.804666 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:17.804583 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" podUID="cda6f83b-4ff5-4028-ab2a-05679c2e6fb8" containerName="payload-processing" containerID="cri-o://b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2" gracePeriod=25 Nov 02 14:53:17.814324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:17.814287 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:18.184054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.184023 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:53:18.295997 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.295959 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-plugins-config-volume\") pod \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\" (UID: \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\") " Nov 02 14:53:18.296199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.295999 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" event={"ID":"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8","Type":"ContainerDied","Data":"b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2"} Nov 02 14:53:18.296199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.296022 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" Nov 02 14:53:18.296199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.296047 5591 scope.go:122] "RemoveContainer" containerID="b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2" Nov 02 14:53:18.296199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.296063 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-kube-api-access-69n55\" (UniqueName: \"kubernetes.io/projected/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-kube-api-access-69n55\") pod \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\" (UID: \"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8\") " Nov 02 14:53:18.296199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.295962 5591 generic.go:356] "Generic (PLEG): container finished" podID="cda6f83b-4ff5-4028-ab2a-05679c2e6fb8" containerID="b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2" exitCode=0 Nov 02 14:53:18.296519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.296276 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp" event={"ID":"cda6f83b-4ff5-4028-ab2a-05679c2e6fb8","Type":"ContainerDied","Data":"16173fb4323c57b7b93af2f97d13dffdbb879d9c8fb7cab9773c84999e9329b4"} Nov 02 14:53:18.296519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.296414 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-plugins-config-volume" pod "cda6f83b-4ff5-4028-ab2a-05679c2e6fb8" (UID: "cda6f83b-4ff5-4028-ab2a-05679c2e6fb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:53:18.296754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.296735 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:18.298390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.298362 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-kube-api-access-69n55" pod "cda6f83b-4ff5-4028-ab2a-05679c2e6fb8" (UID: "cda6f83b-4ff5-4028-ab2a-05679c2e6fb8"). InnerVolumeSpecName "kube-api-access-69n55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:18.314050 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.314028 5591 scope.go:122] "RemoveContainer" containerID="b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2" Nov 02 14:53:18.314337 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:53:18.314313 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2\": container with ID starting with b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2 not found: ID does not exist" containerID="b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2" Nov 02 14:53:18.314389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.314343 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2"} err="failed to get container status \"b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2\": rpc error: code = NotFound desc = could not find container \"b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2\": container with ID starting with b4c58f721835254dbf330d2bce87a8e47bfe2300260a1e13149a61d3b25207a2 not found: ID does not exist" Nov 02 14:53:18.398597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.398565 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69n55\" (UniqueName: \"kubernetes.io/projected/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8-kube-api-access-69n55\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:18.651144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.651100 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp"] Nov 02 14:53:18.664431 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:18.664395 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-662e3e-8469b9bf5-jtwtp"] Nov 02 14:53:19.431867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:19.431827 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda6f83b-4ff5-4028-ab2a-05679c2e6fb8" path="/var/lib/kubelet/pods/cda6f83b-4ff5-4028-ab2a-05679c2e6fb8/volumes" Nov 02 14:53:19.542010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:19.538059 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:19.745468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:19.745383 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:20.493754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:20.493716 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" podUID="1a181a92-b332-4b5a-bc6a-c8208b899b9c" containerName="payload-processing" containerID="cri-o://91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a" gracePeriod=25 Nov 02 14:53:20.773027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:20.772994 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:53:20.853042 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:20.852978 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a181a92-b332-4b5a-bc6a-c8208b899b9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a181a92-b332-4b5a-bc6a-c8208b899b9c-plugins-config-volume\") pod \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\" (UID: \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\") " Nov 02 14:53:20.853212 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:20.853091 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a181a92-b332-4b5a-bc6a-c8208b899b9c-kube-api-access-p5ztf\" (UniqueName: \"kubernetes.io/projected/1a181a92-b332-4b5a-bc6a-c8208b899b9c-kube-api-access-p5ztf\") pod \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\" (UID: \"1a181a92-b332-4b5a-bc6a-c8208b899b9c\") " Nov 02 14:53:20.853508 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:20.853479 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a181a92-b332-4b5a-bc6a-c8208b899b9c-plugins-config-volume" pod "1a181a92-b332-4b5a-bc6a-c8208b899b9c" (UID: "1a181a92-b332-4b5a-bc6a-c8208b899b9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:53:20.855323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:20.855296 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a181a92-b332-4b5a-bc6a-c8208b899b9c-kube-api-access-p5ztf" pod "1a181a92-b332-4b5a-bc6a-c8208b899b9c" (UID: "1a181a92-b332-4b5a-bc6a-c8208b899b9c"). InnerVolumeSpecName "kube-api-access-p5ztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:20.955135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:20.955098 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a181a92-b332-4b5a-bc6a-c8208b899b9c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:20.955135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:20.955124 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ztf\" (UniqueName: \"kubernetes.io/projected/1a181a92-b332-4b5a-bc6a-c8208b899b9c-kube-api-access-p5ztf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:21.210311 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.210252 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:21.317324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.317287 5591 generic.go:356] "Generic (PLEG): container finished" podID="1a181a92-b332-4b5a-bc6a-c8208b899b9c" containerID="91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a" exitCode=0 Nov 02 14:53:21.317543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.317341 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" Nov 02 14:53:21.317543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.317375 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" event={"ID":"1a181a92-b332-4b5a-bc6a-c8208b899b9c","Type":"ContainerDied","Data":"91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a"} Nov 02 14:53:21.317543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.317439 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p" event={"ID":"1a181a92-b332-4b5a-bc6a-c8208b899b9c","Type":"ContainerDied","Data":"9ca011cca25877dfb3b4a6d71904c6ac753d931f56a625f3aa72237e09f0ff0b"} Nov 02 14:53:21.317543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.317463 5591 scope.go:122] "RemoveContainer" containerID="91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a" Nov 02 14:53:21.327747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.327726 5591 scope.go:122] "RemoveContainer" containerID="91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a" Nov 02 14:53:21.328044 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:53:21.328006 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a\": container with ID starting with 91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a not found: ID does not exist" containerID="91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a" Nov 02 14:53:21.328101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.328044 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a"} err="failed to get container status \"91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a\": rpc error: code = NotFound desc = could not find container \"91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a\": container with ID starting with 91b0144f55bf080a661d0ea076553f9cd9c89f5a7cdbcb99619784b2e646e36a not found: ID does not exist" Nov 02 14:53:21.369926 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.369880 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p"] Nov 02 14:53:21.383022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.382975 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2daae4-7bc5f7cd97-bsn6p"] Nov 02 14:53:21.431936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:21.431899 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a181a92-b332-4b5a-bc6a-c8208b899b9c" path="/var/lib/kubelet/pods/1a181a92-b332-4b5a-bc6a-c8208b899b9c/volumes" Nov 02 14:53:22.458119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:22.458074 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:22.786180 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:22.781794 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:24.090572 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:24.090536 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:24.505829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:24.505789 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:26.160132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:26.160087 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:27.854005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:27.844300 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:28.260159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:28.260128 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:29.135963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.135927 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5"] Nov 02 14:53:29.136408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.136194 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" podUID="2223ad11-a345-472c-8254-879383227f0b" containerName="maas-api" containerID="cri-o://66a071d43f243e2748658e2d9521a298e8901cdd4689e92830c59a49459a6993" gracePeriod=30 Nov 02 14:53:29.254399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.254370 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:29.504588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.504556 5591 generic.go:356] "Generic (PLEG): container finished" podID="2223ad11-a345-472c-8254-879383227f0b" containerID="66a071d43f243e2748658e2d9521a298e8901cdd4689e92830c59a49459a6993" exitCode=0 Nov 02 14:53:29.504791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.504704 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" event={"ID":"2223ad11-a345-472c-8254-879383227f0b","Type":"ContainerDied","Data":"66a071d43f243e2748658e2d9521a298e8901cdd4689e92830c59a49459a6993"} Nov 02 14:53:29.531472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.531433 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:53:29.630005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.627513 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls\") pod \"2223ad11-a345-472c-8254-879383227f0b\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " Nov 02 14:53:29.630005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.627596 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-metrics-certs\") pod \"2223ad11-a345-472c-8254-879383227f0b\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " Nov 02 14:53:29.630005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.627699 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2223ad11-a345-472c-8254-879383227f0b-kube-api-access-t7zvh\" (UniqueName: \"kubernetes.io/projected/2223ad11-a345-472c-8254-879383227f0b-kube-api-access-t7zvh\") pod \"2223ad11-a345-472c-8254-879383227f0b\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " Nov 02 14:53:29.630005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.627832 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2223ad11-a345-472c-8254-879383227f0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2223ad11-a345-472c-8254-879383227f0b-openshift-service-ca\") pod \"2223ad11-a345-472c-8254-879383227f0b\" (UID: \"2223ad11-a345-472c-8254-879383227f0b\") " Nov 02 14:53:29.630005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.629103 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2223ad11-a345-472c-8254-879383227f0b-openshift-service-ca" pod "2223ad11-a345-472c-8254-879383227f0b" (UID: "2223ad11-a345-472c-8254-879383227f0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:53:29.640190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.640151 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2223ad11-a345-472c-8254-879383227f0b-kube-api-access-t7zvh" pod "2223ad11-a345-472c-8254-879383227f0b" (UID: "2223ad11-a345-472c-8254-879383227f0b"). InnerVolumeSpecName "kube-api-access-t7zvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:29.641892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.641856 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-metrics-certs" pod "2223ad11-a345-472c-8254-879383227f0b" (UID: "2223ad11-a345-472c-8254-879383227f0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:53:29.643701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.643673 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls" pod "2223ad11-a345-472c-8254-879383227f0b" (UID: "2223ad11-a345-472c-8254-879383227f0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:53:29.729764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.729636 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2223ad11-a345-472c-8254-879383227f0b-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:29.729764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.729677 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:29.729764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.729689 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2223ad11-a345-472c-8254-879383227f0b-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:29.729764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.729697 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zvh\" (UniqueName: \"kubernetes.io/projected/2223ad11-a345-472c-8254-879383227f0b-kube-api-access-t7zvh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:29.880589 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:29.880551 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:30.513881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:30.513801 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" Nov 02 14:53:30.513881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:30.513831 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" event={"ID":"2223ad11-a345-472c-8254-879383227f0b","Type":"ContainerDied","Data":"6c964b3880c5f96056d60cc096d31402fccdf6a335201dd535be52c6ff6c1a8a"} Nov 02 14:53:30.514341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:30.513896 5591 scope.go:122] "RemoveContainer" containerID="66a071d43f243e2748658e2d9521a298e8901cdd4689e92830c59a49459a6993" Nov 02 14:53:30.567521 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:30.567491 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5"] Nov 02 14:53:30.579792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:30.579761 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5"] Nov 02 14:53:31.074662 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:53:31.074626 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:53:31.435055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:31.431934 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:31.458563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:31.458521 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2223ad11-a345-472c-8254-879383227f0b" path="/var/lib/kubelet/pods/2223ad11-a345-472c-8254-879383227f0b/volumes" Nov 02 14:53:32.461250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:32.461197 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2daae4-7cd96c8b6d-2f5c5" podUID="2223ad11-a345-472c-8254-879383227f0b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.171:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:53:33.047919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:33.047801 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:34.110094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:34.110059 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:34.742778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:34.742743 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:36.342205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:36.342158 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:38.194637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:38.194564 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:39.251062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:39.251008 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:39.431790 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:39.431740 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:39.633064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:39.631900 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:39.943619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:39.943531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:40.137693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:40.136559 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:40.449184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:40.449150 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k"] Nov 02 14:53:40.449603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:40.449364 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" containerID="cri-o://76346f0d9af6d0393e0396a952dfd6b724bfc4c8052f949f33b688e193411a4c" gracePeriod=30 Nov 02 14:53:40.514073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:40.513974 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:41.986831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:41.986791 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:53:41.987265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:41.986869 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:53:42.545878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:42.545843 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:43.759343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:43.759302 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:45.623800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.623767 5591 generic.go:356] "Generic (PLEG): container finished" podID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerID="76346f0d9af6d0393e0396a952dfd6b724bfc4c8052f949f33b688e193411a4c" exitCode=0 Nov 02 14:53:45.624154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.623844 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" event={"ID":"4258b1b8-9638-4a7f-973c-8209d717bfd0","Type":"ContainerDied","Data":"76346f0d9af6d0393e0396a952dfd6b724bfc4c8052f949f33b688e193411a4c"} Nov 02 14:53:45.717759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.717687 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.781955 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-kube-api-access-xfn8m\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-kube-api-access-xfn8m\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.782084 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-podinfo\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.782173 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-socket\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.782243 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-certs\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.782277 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-data\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.782328 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-token\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-token\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.782373 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4258b1b8-9638-4a7f-973c-8209d717bfd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4258b1b8-9638-4a7f-973c-8209d717bfd0-istiod-ca-cert\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.782491 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-envoy\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.785135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.782544 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-credential-socket\") pod \"4258b1b8-9638-4a7f-973c-8209d717bfd0\" (UID: \"4258b1b8-9638-4a7f-973c-8209d717bfd0\") " Nov 02 14:53:45.793406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.793277 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-certs" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:45.794137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.794100 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-credential-socket" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:45.794605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.794579 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4258b1b8-9638-4a7f-973c-8209d717bfd0-istiod-ca-cert" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:53:45.794947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.794919 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-data" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:45.804582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.797357 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-socket" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:45.804582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.797681 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:45.804582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.803622 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-podinfo" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:53:45.825138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.824432 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-envoy" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:53:45.825138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.824472 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-kube-api-access-xfn8m" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "kube-api-access-xfn8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:45.825138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.824850 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-token" pod "4258b1b8-9638-4a7f-973c-8209d717bfd0" (UID: "4258b1b8-9638-4a7f-973c-8209d717bfd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:53:45.884307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884275 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.884307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884303 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.884307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884315 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfn8m\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-kube-api-access-xfn8m\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.884544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884324 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.884544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884333 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.884544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884341 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.884544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884349 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.884544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884358 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4258b1b8-9638-4a7f-973c-8209d717bfd0-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.884544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.884366 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4258b1b8-9638-4a7f-973c-8209d717bfd0-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:53:45.952777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:45.952740 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj"] Nov 02 14:53:46.215899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:46.215865 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:46.631387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:46.631353 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" Nov 02 14:53:46.631885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:46.631387 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k" event={"ID":"4258b1b8-9638-4a7f-973c-8209d717bfd0","Type":"ContainerDied","Data":"c84d44fc38f0511fecdcf3c672efa76aa1fc18ab089e5a25651ef1c85f8f98ab"} Nov 02 14:53:46.631885 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:46.631457 5591 scope.go:122] "RemoveContainer" containerID="76346f0d9af6d0393e0396a952dfd6b724bfc4c8052f949f33b688e193411a4c" Nov 02 14:53:46.697677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:46.697637 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k"] Nov 02 14:53:46.704078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:46.704046 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-rvq4k"] Nov 02 14:53:47.431548 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:47.431514 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" path="/var/lib/kubelet/pods/4258b1b8-9638-4a7f-973c-8209d717bfd0/volumes" Nov 02 14:53:47.458434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:47.458245 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:48.719205 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:48.719168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:49.137059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:49.134640 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:50.865294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:50.865236 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:52.461170 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:52.461132 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:52.766325 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:52.766206 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:53:54.374722 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:54.374687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:54.553494 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:54.553443 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:55.353161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:55.353123 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:55.679247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:55.679215 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:56.302915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:56.302882 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:56.578286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:56.578203 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:56.899490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:56.899453 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:57.202412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:57.202332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:58.614701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:58.614665 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr"] Nov 02 14:53:58.653977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:58.653925 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:58.664072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:58.664005 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm"] Nov 02 14:53:58.827767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:58.827707 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:53:58.993007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:53:58.992966 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:00.535006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:00.524579 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:00.974138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:00.974092 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" containerID="cri-o://496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b" gracePeriod=45 Nov 02 14:54:01.073760 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:54:01.073719 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:54:01.247585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.247556 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:54:01.336692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.336652 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-dshm\") pod \"ab68e373-a771-4e18-b873-d46cba1c4232\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " Nov 02 14:54:01.336901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.336712 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab68e373-a771-4e18-b873-d46cba1c4232-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab68e373-a771-4e18-b873-d46cba1c4232-tls-certs\") pod \"ab68e373-a771-4e18-b873-d46cba1c4232\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " Nov 02 14:54:01.336901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.336765 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-tmp-dir\") pod \"ab68e373-a771-4e18-b873-d46cba1c4232\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " Nov 02 14:54:01.336901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.336801 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab68e373-a771-4e18-b873-d46cba1c4232-kube-api-access-zg8hb\" (UniqueName: \"kubernetes.io/projected/ab68e373-a771-4e18-b873-d46cba1c4232-kube-api-access-zg8hb\") pod \"ab68e373-a771-4e18-b873-d46cba1c4232\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " Nov 02 14:54:01.336901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.336886 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-model-cache\") pod \"ab68e373-a771-4e18-b873-d46cba1c4232\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " Nov 02 14:54:01.337199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.337024 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-home\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-home\") pod \"ab68e373-a771-4e18-b873-d46cba1c4232\" (UID: \"ab68e373-a771-4e18-b873-d46cba1c4232\") " Nov 02 14:54:01.337199 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.337154 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-tmp-dir" pod "ab68e373-a771-4e18-b873-d46cba1c4232" (UID: "ab68e373-a771-4e18-b873-d46cba1c4232"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:01.337528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.337340 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-model-cache" pod "ab68e373-a771-4e18-b873-d46cba1c4232" (UID: "ab68e373-a771-4e18-b873-d46cba1c4232"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:01.337528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.337441 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-home" pod "ab68e373-a771-4e18-b873-d46cba1c4232" (UID: "ab68e373-a771-4e18-b873-d46cba1c4232"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:01.337732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.337647 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:01.337732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.337672 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:01.337732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.337686 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:01.341862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.341833 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab68e373-a771-4e18-b873-d46cba1c4232-tls-certs" pod "ab68e373-a771-4e18-b873-d46cba1c4232" (UID: "ab68e373-a771-4e18-b873-d46cba1c4232"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:54:01.342143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.342115 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-dshm" pod "ab68e373-a771-4e18-b873-d46cba1c4232" (UID: "ab68e373-a771-4e18-b873-d46cba1c4232"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:01.342269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.342118 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab68e373-a771-4e18-b873-d46cba1c4232-kube-api-access-zg8hb" pod "ab68e373-a771-4e18-b873-d46cba1c4232" (UID: "ab68e373-a771-4e18-b873-d46cba1c4232"). InnerVolumeSpecName "kube-api-access-zg8hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:01.439860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.439820 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab68e373-a771-4e18-b873-d46cba1c4232-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:01.439860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.439860 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab68e373-a771-4e18-b873-d46cba1c4232-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:01.440159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.439876 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg8hb\" (UniqueName: \"kubernetes.io/projected/ab68e373-a771-4e18-b873-d46cba1c4232-kube-api-access-zg8hb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:01.796901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.796862 5591 generic.go:356] "Generic (PLEG): container finished" podID="ab68e373-a771-4e18-b873-d46cba1c4232" containerID="496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b" exitCode=0 Nov 02 14:54:01.796901 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.796906 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" Nov 02 14:54:01.797430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.796953 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" event={"ID":"ab68e373-a771-4e18-b873-d46cba1c4232","Type":"ContainerDied","Data":"496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b"} Nov 02 14:54:01.797430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.797038 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj" event={"ID":"ab68e373-a771-4e18-b873-d46cba1c4232","Type":"ContainerDied","Data":"e324376604d7cb87473c9a6bd64117e6cf5a537106734b629eda2aabbb835688"} Nov 02 14:54:01.797430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.797065 5591 scope.go:122] "RemoveContainer" containerID="496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b" Nov 02 14:54:01.806862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.806838 5591 scope.go:122] "RemoveContainer" containerID="496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b" Nov 02 14:54:01.807243 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:54:01.807216 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b\": container with ID starting with 496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b not found: ID does not exist" containerID="496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b" Nov 02 14:54:01.807323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.807257 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b"} err="failed to get container status \"496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b\": rpc error: code = NotFound desc = could not find container \"496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b\": container with ID starting with 496f15f275e14fc3417ecc38153d5c93e26ea609e872ecfcdebb1a0cf8b02f3b not found: ID does not exist" Nov 02 14:54:01.848908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.848867 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj"] Nov 02 14:54:01.860668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:01.860634 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/simulator-w0-1a562c-kserve-577bc7ccdf-7hvrj"] Nov 02 14:54:02.124939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:02.124787 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:02.135752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:02.135693 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl"] Nov 02 14:54:02.135963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:02.135766 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc"] Nov 02 14:54:02.140495 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:02.140455 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk"] Nov 02 14:54:02.373886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:02.373848 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:03.058937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:03.058894 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:03.431759 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:03.431726 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" path="/var/lib/kubelet/pods/ab68e373-a771-4e18-b873-d46cba1c4232/volumes" Nov 02 14:54:03.615145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:03.615098 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" podUID="9fe3bb4e-5a5c-4019-aada-a451a33c84c6" containerName="payload-processing" containerID="cri-o://d6976699aaaedc4b4aa3013bd6461cf40e35cad2b3614f68991f428fa24559ea" gracePeriod=25 Nov 02 14:54:03.665915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:03.665853 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" podUID="5fe100e2-966f-4c81-b6d6-41347f79710f" containerName="payload-pre-processing" containerID="cri-o://825ce3a2dd8f072fb5f41b3131f5888138a2f2e5e9a5c598321f78364c966d13" gracePeriod=25 Nov 02 14:54:03.844818 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:03.844726 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:05.240723 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:05.240687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:05.570480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:05.570391 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:07.170008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:07.167173 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:08.775776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:08.775739 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:08.983075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:08.983037 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7"] Nov 02 14:54:09.129785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:09.129696 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:09.330341 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:09.330297 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:10.243272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:10.243239 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:10.640915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:10.636072 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:11.469051 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:11.469013 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:12.081321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:12.081278 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" podUID="9fe3bb4e-5a5c-4019-aada-a451a33c84c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:54:12.081514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:12.081463 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" podUID="5fe100e2-966f-4c81-b6d6-41347f79710f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:54:12.271155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:12.271114 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:12.297429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:12.297401 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:54:12.319258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:12.319229 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:54:13.987027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:13.983644 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" podUID="e6f0d23b-ef8c-4f01-a66d-622d13eccd48" containerName="payload-pre-processing" containerID="cri-o://ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953" gracePeriod=25 Nov 02 14:54:14.004421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:14.004218 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:15.777971 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:15.777900 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:16.570418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:16.570383 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:17.157126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.157077 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" containerID="cri-o://6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523" gracePeriod=45 Nov 02 14:54:17.157806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.157622 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" containerID="cri-o://9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1" gracePeriod=45 Nov 02 14:54:17.159652 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.159621 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" containerID="cri-o://f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8" gracePeriod=45 Nov 02 14:54:17.499007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.498962 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:54:17.508797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.508755 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:54:17.513320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.513294 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:54:17.549891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.549849 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:17.650725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650421 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-dshm\") pod \"f4d6146c-41ab-41c7-a5f5-222f67358b69\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " Nov 02 14:54:17.650725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650481 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-kube-api-access-lcssr\" (UniqueName: \"kubernetes.io/projected/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-kube-api-access-lcssr\") pod \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " Nov 02 14:54:17.650725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650518 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-model-cache\") pod \"bc692aee-a900-41cb-82ef-911d588161f3\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " Nov 02 14:54:17.650725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650594 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tmp-dir\") pod \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " Nov 02 14:54:17.650725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650626 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-dshm\") pod \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " Nov 02 14:54:17.650725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650652 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-home\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-home\") pod \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " Nov 02 14:54:17.650725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650684 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-model-cache\") pod \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " Nov 02 14:54:17.650725 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650721 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc692aee-a900-41cb-82ef-911d588161f3-kube-api-access-87t5t\" (UniqueName: \"kubernetes.io/projected/bc692aee-a900-41cb-82ef-911d588161f3-kube-api-access-87t5t\") pod \"bc692aee-a900-41cb-82ef-911d588161f3\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650757 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-tmp-dir\") pod \"bc692aee-a900-41cb-82ef-911d588161f3\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650787 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc692aee-a900-41cb-82ef-911d588161f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc692aee-a900-41cb-82ef-911d588161f3-tls-certs\") pod \"bc692aee-a900-41cb-82ef-911d588161f3\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650834 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d6146c-41ab-41c7-a5f5-222f67358b69-kube-api-access-84lcx\" (UniqueName: \"kubernetes.io/projected/f4d6146c-41ab-41c7-a5f5-222f67358b69-kube-api-access-84lcx\") pod \"f4d6146c-41ab-41c7-a5f5-222f67358b69\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650871 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-home\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-home\") pod \"f4d6146c-41ab-41c7-a5f5-222f67358b69\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650901 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-home\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-home\") pod \"bc692aee-a900-41cb-82ef-911d588161f3\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.650996 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-dshm\") pod \"bc692aee-a900-41cb-82ef-911d588161f3\" (UID: \"bc692aee-a900-41cb-82ef-911d588161f3\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651105 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tls-certs\") pod \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\" (UID: \"ea87f45c-22b3-4eb0-a1f9-47bc7c795155\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651108 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-model-cache" pod "ea87f45c-22b3-4eb0-a1f9-47bc7c795155" (UID: "ea87f45c-22b3-4eb0-a1f9-47bc7c795155"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651146 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d6146c-41ab-41c7-a5f5-222f67358b69-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d6146c-41ab-41c7-a5f5-222f67358b69-tls-certs\") pod \"f4d6146c-41ab-41c7-a5f5-222f67358b69\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651196 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-model-cache\") pod \"f4d6146c-41ab-41c7-a5f5-222f67358b69\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651292 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-tmp-dir\") pod \"f4d6146c-41ab-41c7-a5f5-222f67358b69\" (UID: \"f4d6146c-41ab-41c7-a5f5-222f67358b69\") " Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651348 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-model-cache" pod "bc692aee-a900-41cb-82ef-911d588161f3" (UID: "bc692aee-a900-41cb-82ef-911d588161f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651567 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tmp-dir" pod "ea87f45c-22b3-4eb0-a1f9-47bc7c795155" (UID: "ea87f45c-22b3-4eb0-a1f9-47bc7c795155"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.651822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651798 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-tmp-dir" pod "f4d6146c-41ab-41c7-a5f5-222f67358b69" (UID: "f4d6146c-41ab-41c7-a5f5-222f67358b69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.652766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651895 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.652766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651916 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.652766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.651930 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.652766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.652041 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-home" pod "ea87f45c-22b3-4eb0-a1f9-47bc7c795155" (UID: "ea87f45c-22b3-4eb0-a1f9-47bc7c795155"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.652766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.652148 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-home" pod "f4d6146c-41ab-41c7-a5f5-222f67358b69" (UID: "f4d6146c-41ab-41c7-a5f5-222f67358b69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.652766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.652323 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-home" pod "bc692aee-a900-41cb-82ef-911d588161f3" (UID: "bc692aee-a900-41cb-82ef-911d588161f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.654284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.654217 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tls-certs" pod "ea87f45c-22b3-4eb0-a1f9-47bc7c795155" (UID: "ea87f45c-22b3-4eb0-a1f9-47bc7c795155"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:54:17.654485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.654460 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-model-cache" pod "f4d6146c-41ab-41c7-a5f5-222f67358b69" (UID: "f4d6146c-41ab-41c7-a5f5-222f67358b69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.655055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.655031 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-tmp-dir" pod "bc692aee-a900-41cb-82ef-911d588161f3" (UID: "bc692aee-a900-41cb-82ef-911d588161f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.656148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.656124 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-dshm" pod "f4d6146c-41ab-41c7-a5f5-222f67358b69" (UID: "f4d6146c-41ab-41c7-a5f5-222f67358b69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.656148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.656136 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d6146c-41ab-41c7-a5f5-222f67358b69-tls-certs" pod "f4d6146c-41ab-41c7-a5f5-222f67358b69" (UID: "f4d6146c-41ab-41c7-a5f5-222f67358b69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:54:17.656373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.656355 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-kube-api-access-lcssr" pod "ea87f45c-22b3-4eb0-a1f9-47bc7c795155" (UID: "ea87f45c-22b3-4eb0-a1f9-47bc7c795155"). InnerVolumeSpecName "kube-api-access-lcssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:17.656800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.656783 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc692aee-a900-41cb-82ef-911d588161f3-kube-api-access-87t5t" pod "bc692aee-a900-41cb-82ef-911d588161f3" (UID: "bc692aee-a900-41cb-82ef-911d588161f3"). InnerVolumeSpecName "kube-api-access-87t5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:17.657316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.657293 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d6146c-41ab-41c7-a5f5-222f67358b69-kube-api-access-84lcx" pod "f4d6146c-41ab-41c7-a5f5-222f67358b69" (UID: "f4d6146c-41ab-41c7-a5f5-222f67358b69"). InnerVolumeSpecName "kube-api-access-84lcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:17.657434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.657312 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-dshm" pod "ea87f45c-22b3-4eb0-a1f9-47bc7c795155" (UID: "ea87f45c-22b3-4eb0-a1f9-47bc7c795155"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.657919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.657899 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc692aee-a900-41cb-82ef-911d588161f3-tls-certs" pod "bc692aee-a900-41cb-82ef-911d588161f3" (UID: "bc692aee-a900-41cb-82ef-911d588161f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:54:17.658749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.658722 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-dshm" pod "bc692aee-a900-41cb-82ef-911d588161f3" (UID: "bc692aee-a900-41cb-82ef-911d588161f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754007 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754052 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754068 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87t5t\" (UniqueName: \"kubernetes.io/projected/bc692aee-a900-41cb-82ef-911d588161f3-kube-api-access-87t5t\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754084 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754099 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc692aee-a900-41cb-82ef-911d588161f3-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754169 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84lcx\" (UniqueName: \"kubernetes.io/projected/f4d6146c-41ab-41c7-a5f5-222f67358b69-kube-api-access-84lcx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754181 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754196 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754208 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc692aee-a900-41cb-82ef-911d588161f3-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754222 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754235 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4d6146c-41ab-41c7-a5f5-222f67358b69-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754247 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754261 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754273 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4d6146c-41ab-41c7-a5f5-222f67358b69-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.756081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.754285 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcssr\" (UniqueName: \"kubernetes.io/projected/ea87f45c-22b3-4eb0-a1f9-47bc7c795155-kube-api-access-lcssr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:17.907602 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.907566 5591 generic.go:356] "Generic (PLEG): container finished" podID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerID="9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1" exitCode=0 Nov 02 14:54:17.907820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.907805 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" Nov 02 14:54:17.908568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.908535 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" event={"ID":"ea87f45c-22b3-4eb0-a1f9-47bc7c795155","Type":"ContainerDied","Data":"9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1"} Nov 02 14:54:17.908695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.908588 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl" event={"ID":"ea87f45c-22b3-4eb0-a1f9-47bc7c795155","Type":"ContainerDied","Data":"bd0a4b5e50a74bcd41ea1a3191dbf7a106512a6115c260992439bd63f0903814"} Nov 02 14:54:17.908695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.908612 5591 scope.go:122] "RemoveContainer" containerID="9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1" Nov 02 14:54:17.910421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.910395 5591 generic.go:356] "Generic (PLEG): container finished" podID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerID="6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523" exitCode=0 Nov 02 14:54:17.910537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.910475 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" event={"ID":"f4d6146c-41ab-41c7-a5f5-222f67358b69","Type":"ContainerDied","Data":"6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523"} Nov 02 14:54:17.910537 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.910505 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" event={"ID":"f4d6146c-41ab-41c7-a5f5-222f67358b69","Type":"ContainerDied","Data":"42545f2054cdef83310c46e548b35a321b4ff377a256a061ea07fbb8159cca39"} Nov 02 14:54:17.910673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.910576 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc" Nov 02 14:54:17.913490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.913461 5591 generic.go:356] "Generic (PLEG): container finished" podID="bc692aee-a900-41cb-82ef-911d588161f3" containerID="f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8" exitCode=0 Nov 02 14:54:17.913623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.913507 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" event={"ID":"bc692aee-a900-41cb-82ef-911d588161f3","Type":"ContainerDied","Data":"f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8"} Nov 02 14:54:17.913623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.913536 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" event={"ID":"bc692aee-a900-41cb-82ef-911d588161f3","Type":"ContainerDied","Data":"9ab0f95536b5bd79c7a7d20660aebacfe886bbf87970ff6141c84ad34aa25952"} Nov 02 14:54:17.913623 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.913600 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" Nov 02 14:54:17.923957 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.923928 5591 scope.go:122] "RemoveContainer" containerID="9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1" Nov 02 14:54:17.924360 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:54:17.924333 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1\": container with ID starting with 9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1 not found: ID does not exist" containerID="9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1" Nov 02 14:54:17.924468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.924366 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1"} err="failed to get container status \"9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1\": rpc error: code = NotFound desc = could not find container \"9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1\": container with ID starting with 9bd27872cbffdc15ec0e33d1d15c0fc72929b1848758b50c4851c6bc6041bfb1 not found: ID does not exist" Nov 02 14:54:17.924468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.924382 5591 scope.go:122] "RemoveContainer" containerID="6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523" Nov 02 14:54:17.938697 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.938671 5591 scope.go:122] "RemoveContainer" containerID="6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523" Nov 02 14:54:17.939046 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:54:17.939023 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523\": container with ID starting with 6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523 not found: ID does not exist" containerID="6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523" Nov 02 14:54:17.939386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.939053 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523"} err="failed to get container status \"6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523\": rpc error: code = NotFound desc = could not find container \"6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523\": container with ID starting with 6b6b14a0be1253506cb8587f7679e09023076fe8f7237eff4bccf73127255523 not found: ID does not exist" Nov 02 14:54:17.939386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.939070 5591 scope.go:122] "RemoveContainer" containerID="f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8" Nov 02 14:54:17.948078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.948056 5591 scope.go:122] "RemoveContainer" containerID="f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8" Nov 02 14:54:17.948327 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:54:17.948306 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8\": container with ID starting with f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8 not found: ID does not exist" containerID="f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8" Nov 02 14:54:17.948395 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.948336 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8"} err="failed to get container status \"f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8\": rpc error: code = NotFound desc = could not find container \"f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8\": container with ID starting with f518b1881839aca7d8b7168d90722aefc106afc0190bc0a131d1609a0d918ce8 not found: ID does not exist" Nov 02 14:54:17.967321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.967294 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl"] Nov 02 14:54:17.975600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:17.975577 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-w3-f30173-kserve-766464cb68-p2bxl"] Nov 02 14:54:18.038814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:18.038729 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk"] Nov 02 14:54:18.042745 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:18.042719 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk"] Nov 02 14:54:18.079099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:18.079067 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc"] Nov 02 14:54:18.082785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:18.082759 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/premium-w3-f30173-kserve-6b5d65cbb5-wqlqc"] Nov 02 14:54:18.117908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:18.117876 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" podUID="e6f0d23b-ef8c-4f01-a66d-622d13eccd48" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:54:18.475396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:18.475332 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/simulator-w3-f30173-kserve-5c54fbb6c6-96cpk" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 02 14:54:19.120829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:19.120796 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:19.536835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:19.535142 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc692aee-a900-41cb-82ef-911d588161f3" path="/var/lib/kubelet/pods/bc692aee-a900-41cb-82ef-911d588161f3/volumes" Nov 02 14:54:19.536835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:19.535696 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" path="/var/lib/kubelet/pods/ea87f45c-22b3-4eb0-a1f9-47bc7c795155/volumes" Nov 02 14:54:19.537313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:19.536957 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" path="/var/lib/kubelet/pods/f4d6146c-41ab-41c7-a5f5-222f67358b69/volumes" Nov 02 14:54:19.873456 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:19.873360 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:21.553320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:21.553280 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:22.081323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:22.081271 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" podUID="9fe3bb4e-5a5c-4019-aada-a451a33c84c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:54:22.081567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:22.081542 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" podUID="5fe100e2-966f-4c81-b6d6-41347f79710f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:54:23.086019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:23.073713 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:23.278867 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:23.278815 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:23.449795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:23.449763 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:23.960386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:23.960296 5591 generic.go:356] "Generic (PLEG): container finished" podID="9fe3bb4e-5a5c-4019-aada-a451a33c84c6" containerID="d6976699aaaedc4b4aa3013bd6461cf40e35cad2b3614f68991f428fa24559ea" exitCode=0 Nov 02 14:54:23.960386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:23.960371 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" event={"ID":"9fe3bb4e-5a5c-4019-aada-a451a33c84c6","Type":"ContainerDied","Data":"d6976699aaaedc4b4aa3013bd6461cf40e35cad2b3614f68991f428fa24559ea"} Nov 02 14:54:23.962105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:23.962069 5591 generic.go:356] "Generic (PLEG): container finished" podID="5fe100e2-966f-4c81-b6d6-41347f79710f" containerID="825ce3a2dd8f072fb5f41b3131f5888138a2f2e5e9a5c598321f78364c966d13" exitCode=0 Nov 02 14:54:23.962254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:23.962175 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" event={"ID":"5fe100e2-966f-4c81-b6d6-41347f79710f","Type":"ContainerDied","Data":"825ce3a2dd8f072fb5f41b3131f5888138a2f2e5e9a5c598321f78364c966d13"} Nov 02 14:54:24.035543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.035515 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 14:54:24.078587 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.078562 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 14:54:24.175213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.175181 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-kube-api-access-dj76f\" (UniqueName: \"kubernetes.io/projected/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-kube-api-access-dj76f\") pod \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\" (UID: \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\") " Nov 02 14:54:24.175762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.175249 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe100e2-966f-4c81-b6d6-41347f79710f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe100e2-966f-4c81-b6d6-41347f79710f-plugins-config-volume\") pod \"5fe100e2-966f-4c81-b6d6-41347f79710f\" (UID: \"5fe100e2-966f-4c81-b6d6-41347f79710f\") " Nov 02 14:54:24.175762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.175378 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe100e2-966f-4c81-b6d6-41347f79710f-kube-api-access-6vsg5\" (UniqueName: \"kubernetes.io/projected/5fe100e2-966f-4c81-b6d6-41347f79710f-kube-api-access-6vsg5\") pod \"5fe100e2-966f-4c81-b6d6-41347f79710f\" (UID: \"5fe100e2-966f-4c81-b6d6-41347f79710f\") " Nov 02 14:54:24.175762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.175410 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-plugins-config-volume\") pod \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\" (UID: \"9fe3bb4e-5a5c-4019-aada-a451a33c84c6\") " Nov 02 14:54:24.175951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.175793 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe100e2-966f-4c81-b6d6-41347f79710f-plugins-config-volume" pod "5fe100e2-966f-4c81-b6d6-41347f79710f" (UID: "5fe100e2-966f-4c81-b6d6-41347f79710f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:54:24.175951 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.175851 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-plugins-config-volume" pod "9fe3bb4e-5a5c-4019-aada-a451a33c84c6" (UID: "9fe3bb4e-5a5c-4019-aada-a451a33c84c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:54:24.177527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.177504 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-kube-api-access-dj76f" pod "9fe3bb4e-5a5c-4019-aada-a451a33c84c6" (UID: "9fe3bb4e-5a5c-4019-aada-a451a33c84c6"). InnerVolumeSpecName "kube-api-access-dj76f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:24.177660 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.177561 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe100e2-966f-4c81-b6d6-41347f79710f-kube-api-access-6vsg5" pod "5fe100e2-966f-4c81-b6d6-41347f79710f" (UID: "5fe100e2-966f-4c81-b6d6-41347f79710f"). InnerVolumeSpecName "kube-api-access-6vsg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:24.277086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.276999 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vsg5\" (UniqueName: \"kubernetes.io/projected/5fe100e2-966f-4c81-b6d6-41347f79710f-kube-api-access-6vsg5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:24.277086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.277036 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:24.277086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.277046 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj76f\" (UniqueName: \"kubernetes.io/projected/9fe3bb4e-5a5c-4019-aada-a451a33c84c6-kube-api-access-dj76f\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:24.277086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.277055 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe100e2-966f-4c81-b6d6-41347f79710f-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:24.971020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.970899 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" Nov 02 14:54:24.971020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.970911 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr" event={"ID":"9fe3bb4e-5a5c-4019-aada-a451a33c84c6","Type":"ContainerDied","Data":"1742b9fc9a8cfad4e2eec5ec3da73c86802c339d49e200629f57cc28411881b4"} Nov 02 14:54:24.971020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.971018 5591 scope.go:122] "RemoveContainer" containerID="d6976699aaaedc4b4aa3013bd6461cf40e35cad2b3614f68991f428fa24559ea" Nov 02 14:54:24.972583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.972562 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" Nov 02 14:54:24.972664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.972605 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm" event={"ID":"5fe100e2-966f-4c81-b6d6-41347f79710f","Type":"ContainerDied","Data":"b97b0652f182dd27b615a20dc9e4a0404bbc7671b14bc30ab9e14c0a82bb3ef0"} Nov 02 14:54:24.983788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:24.983759 5591 scope.go:122] "RemoveContainer" containerID="825ce3a2dd8f072fb5f41b3131f5888138a2f2e5e9a5c598321f78364c966d13" Nov 02 14:54:25.027246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:25.027213 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm"] Nov 02 14:54:25.037299 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:25.037271 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a562c-56589b9f59-rmqhm"] Nov 02 14:54:25.062506 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:25.062468 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr"] Nov 02 14:54:25.068669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:25.068631 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a562c-fc4f6d887-xh5kr"] Nov 02 14:54:25.184059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:25.184025 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:25.430866 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:25.430829 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe100e2-966f-4c81-b6d6-41347f79710f" path="/var/lib/kubelet/pods/5fe100e2-966f-4c81-b6d6-41347f79710f/volumes" Nov 02 14:54:25.431344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:25.431327 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe3bb4e-5a5c-4019-aada-a451a33c84c6" path="/var/lib/kubelet/pods/9fe3bb4e-5a5c-4019-aada-a451a33c84c6/volumes" Nov 02 14:54:25.578168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:25.578117 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:26.877014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:26.874184 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:28.130578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:28.130160 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" podUID="e6f0d23b-ef8c-4f01-a66d-622d13eccd48" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:54:28.311567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:28.311528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:28.521182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:28.521147 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:28.911806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:28.911769 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:30.355564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.355526 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2"] Nov 02 14:54:30.356099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.355797 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" podUID="3bead4e2-f184-49b9-8af8-36c25b7bc7dc" containerName="maas-api" containerID="cri-o://fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721" gracePeriod=30 Nov 02 14:54:30.480945 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.480917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:30.713719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.713694 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 14:54:30.779062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.770716 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-openshift-service-ca\") pod \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " Nov 02 14:54:30.779062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.770831 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-metrics-certs\") pod \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " Nov 02 14:54:30.779062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.771049 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-kube-api-access-l78qp\" (UniqueName: \"kubernetes.io/projected/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-kube-api-access-l78qp\") pod \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " Nov 02 14:54:30.779062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.771138 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls\") pod \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\" (UID: \"3bead4e2-f184-49b9-8af8-36c25b7bc7dc\") " Nov 02 14:54:30.863025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.822447 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-openshift-service-ca" pod "3bead4e2-f184-49b9-8af8-36c25b7bc7dc" (UID: "3bead4e2-f184-49b9-8af8-36c25b7bc7dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:54:30.863025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.823431 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-metrics-certs" pod "3bead4e2-f184-49b9-8af8-36c25b7bc7dc" (UID: "3bead4e2-f184-49b9-8af8-36c25b7bc7dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:54:30.863025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.859629 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-kube-api-access-l78qp" pod "3bead4e2-f184-49b9-8af8-36c25b7bc7dc" (UID: "3bead4e2-f184-49b9-8af8-36c25b7bc7dc"). InnerVolumeSpecName "kube-api-access-l78qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:30.863025 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.861090 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls" pod "3bead4e2-f184-49b9-8af8-36c25b7bc7dc" (UID: "3bead4e2-f184-49b9-8af8-36c25b7bc7dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:54:30.878005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.872867 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:30.878005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.872893 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:30.878005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.872903 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:30.878005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:30.872913 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l78qp\" (UniqueName: \"kubernetes.io/projected/3bead4e2-f184-49b9-8af8-36c25b7bc7dc-kube-api-access-l78qp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:31.013272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.013167 5591 generic.go:356] "Generic (PLEG): container finished" podID="3bead4e2-f184-49b9-8af8-36c25b7bc7dc" containerID="fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721" exitCode=0 Nov 02 14:54:31.013272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.013220 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" Nov 02 14:54:31.013513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.013265 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" event={"ID":"3bead4e2-f184-49b9-8af8-36c25b7bc7dc","Type":"ContainerDied","Data":"fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721"} Nov 02 14:54:31.013513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.013333 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2" event={"ID":"3bead4e2-f184-49b9-8af8-36c25b7bc7dc","Type":"ContainerDied","Data":"f217fbaf8da0064a3bb1ccbd8e931b9af9c2ae0aafb1f099254f57c82edd76cc"} Nov 02 14:54:31.013513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.013364 5591 scope.go:122] "RemoveContainer" containerID="fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721" Nov 02 14:54:31.023748 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.023727 5591 scope.go:122] "RemoveContainer" containerID="fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721" Nov 02 14:54:31.024055 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:54:31.024030 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721\": container with ID starting with fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721 not found: ID does not exist" containerID="fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721" Nov 02 14:54:31.024143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.024061 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721"} err="failed to get container status \"fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721\": rpc error: code = NotFound desc = could not find container \"fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721\": container with ID starting with fe2dda389cb5424e7746466d0bf52d61356e2c5f0902080aacaec5c38e113721 not found: ID does not exist" Nov 02 14:54:31.068356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.068321 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2"] Nov 02 14:54:31.074454 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:54:31.074408 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:54:31.079409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.079374 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f30173-7f78958f8d-xfsq2"] Nov 02 14:54:31.163520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.163486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:31.431771 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:31.431736 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bead4e2-f184-49b9-8af8-36c25b7bc7dc" path="/var/lib/kubelet/pods/3bead4e2-f184-49b9-8af8-36c25b7bc7dc/volumes" Nov 02 14:54:32.673214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:32.673175 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:32.879553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:32.879510 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:33.082135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:33.082049 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:33.652412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:33.652359 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:33.845179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:33.845145 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:34.246939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:34.246913 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 14:54:34.313335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:34.313240 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-kube-api-access-rjf7d\" (UniqueName: \"kubernetes.io/projected/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-kube-api-access-rjf7d\") pod \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\" (UID: \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\") " Nov 02 14:54:34.313658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:34.313476 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-plugins-config-volume\") pod \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\" (UID: \"e6f0d23b-ef8c-4f01-a66d-622d13eccd48\") " Nov 02 14:54:34.314198 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:34.314172 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-plugins-config-volume" pod "e6f0d23b-ef8c-4f01-a66d-622d13eccd48" (UID: "e6f0d23b-ef8c-4f01-a66d-622d13eccd48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:54:34.316699 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:34.316665 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-kube-api-access-rjf7d" pod "e6f0d23b-ef8c-4f01-a66d-622d13eccd48" (UID: "e6f0d23b-ef8c-4f01-a66d-622d13eccd48"). InnerVolumeSpecName "kube-api-access-rjf7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:34.415517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:34.415481 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:34.415517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:34.415517 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjf7d\" (UniqueName: \"kubernetes.io/projected/e6f0d23b-ef8c-4f01-a66d-622d13eccd48-kube-api-access-rjf7d\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:34.711332 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:34.711216 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:35.041427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.041337 5591 generic.go:356] "Generic (PLEG): container finished" podID="e6f0d23b-ef8c-4f01-a66d-622d13eccd48" containerID="ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953" exitCode=0 Nov 02 14:54:35.041427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.041384 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" Nov 02 14:54:35.041923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.041428 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" event={"ID":"e6f0d23b-ef8c-4f01-a66d-622d13eccd48","Type":"ContainerDied","Data":"ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953"} Nov 02 14:54:35.041923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.041496 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7" event={"ID":"e6f0d23b-ef8c-4f01-a66d-622d13eccd48","Type":"ContainerDied","Data":"7b4e1edb049291c4f80354f037d0fa64e65671926f4618861d5b8f9748c7f82a"} Nov 02 14:54:35.041923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.041530 5591 scope.go:122] "RemoveContainer" containerID="ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953" Nov 02 14:54:35.054284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.054257 5591 scope.go:122] "RemoveContainer" containerID="ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953" Nov 02 14:54:35.054616 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:54:35.054587 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953\": container with ID starting with ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953 not found: ID does not exist" containerID="ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953" Nov 02 14:54:35.054658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.054629 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953"} err="failed to get container status \"ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953\": rpc error: code = NotFound desc = could not find container \"ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953\": container with ID starting with ba958de09ce450aeda71b32fafb0761f70388ff734f974276e17f9df9eae1953 not found: ID does not exist" Nov 02 14:54:35.094220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.094186 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7"] Nov 02 14:54:35.106275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.106244 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f30173-545fcb4b9f-jpcv7"] Nov 02 14:54:35.431377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:35.431341 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f0d23b-ef8c-4f01-a66d-622d13eccd48" path="/var/lib/kubelet/pods/e6f0d23b-ef8c-4f01-a66d-622d13eccd48/volumes" Nov 02 14:54:36.567862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.564708 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:36.953109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953068 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng"] Nov 02 14:54:36.953744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953716 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" Nov 02 14:54:36.953744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953742 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953762 5591 cpu_manager.go:414] "containerMap: removing container" podUID="cda6f83b-4ff5-4028-ab2a-05679c2e6fb8" containerName="payload-processing" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953770 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda6f83b-4ff5-4028-ab2a-05679c2e6fb8" containerName="payload-processing" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953781 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2223ad11-a345-472c-8254-879383227f0b" containerName="maas-api" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953789 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2223ad11-a345-472c-8254-879383227f0b" containerName="maas-api" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953805 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5fe100e2-966f-4c81-b6d6-41347f79710f" containerName="payload-pre-processing" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953812 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe100e2-966f-4c81-b6d6-41347f79710f" containerName="payload-pre-processing" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953824 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953831 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953840 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953848 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953862 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953870 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953884 5591 cpu_manager.go:414] "containerMap: removing container" podUID="17d10963-ca4b-4b84-ae31-a8b9981f78b3" containerName="payload-pre-processing" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953890 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d10963-ca4b-4b84-ae31-a8b9981f78b3" containerName="payload-pre-processing" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953905 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953911 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953927 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7a879895-0675-4c59-a0f4-ab94b9846d47" containerName="payload-pre-processing" Nov 02 14:54:36.953928 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953934 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a879895-0675-4c59-a0f4-ab94b9846d47" containerName="payload-pre-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953950 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953957 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953967 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.953973 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954002 5591 cpu_manager.go:414] "containerMap: removing container" podUID="95c695c6-5acc-481e-a4b0-06c5fe2d933b" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954010 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c695c6-5acc-481e-a4b0-06c5fe2d933b" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954019 5591 cpu_manager.go:414] "containerMap: removing container" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954026 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954041 5591 cpu_manager.go:414] "containerMap: removing container" podUID="9d6b9d59-c7f6-4f43-b2b7-85137885ec3a" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954048 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6b9d59-c7f6-4f43-b2b7-85137885ec3a" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954060 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e6f0d23b-ef8c-4f01-a66d-622d13eccd48" containerName="payload-pre-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954067 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f0d23b-ef8c-4f01-a66d-622d13eccd48" containerName="payload-pre-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954078 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3bead4e2-f184-49b9-8af8-36c25b7bc7dc" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954086 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bead4e2-f184-49b9-8af8-36c25b7bc7dc" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954097 5591 cpu_manager.go:414] "containerMap: removing container" podUID="cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954104 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954114 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1a181a92-b332-4b5a-bc6a-c8208b899b9c" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954121 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a181a92-b332-4b5a-bc6a-c8208b899b9c" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954133 5591 cpu_manager.go:414] "containerMap: removing container" podUID="9fe3bb4e-5a5c-4019-aada-a451a33c84c6" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954140 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe3bb4e-5a5c-4019-aada-a451a33c84c6" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954161 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c30329ab-6930-4243-826e-4f1733f6172a" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954168 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30329ab-6930-4243-826e-4f1733f6172a" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954184 5591 cpu_manager.go:414] "containerMap: removing container" podUID="36dd4024-c8de-4906-a1c6-cc0a0e757574" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954190 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dd4024-c8de-4906-a1c6-cc0a0e757574" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954206 5591 cpu_manager.go:414] "containerMap: removing container" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954217 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954241 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d33bbdce-378e-4067-8323-3a1a9892f2e6" containerName="payload-pre-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954249 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33bbdce-378e-4067-8323-3a1a9892f2e6" containerName="payload-pre-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954418 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30329ab-6930-4243-826e-4f1733f6172a" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954431 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dd4024-c8de-4906-a1c6-cc0a0e757574" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954446 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954456 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954469 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33bbdce-378e-4067-8323-3a1a9892f2e6" containerName="payload-pre-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954479 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954488 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda6f83b-4ff5-4028-ab2a-05679c2e6fb8" containerName="payload-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954495 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2223ad11-a345-472c-8254-879383227f0b" containerName="maas-api" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954506 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe100e2-966f-4c81-b6d6-41347f79710f" containerName="payload-pre-processing" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954516 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc692aee-a900-41cb-82ef-911d588161f3" containerName="main" Nov 02 14:54:36.954514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954526 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d6146c-41ab-41c7-a5f5-222f67358b69" containerName="main" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954534 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954544 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d10963-ca4b-4b84-ae31-a8b9981f78b3" containerName="payload-pre-processing" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954554 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab68e373-a771-4e18-b873-d46cba1c4232" containerName="main" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954564 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a879895-0675-4c59-a0f4-ab94b9846d47" containerName="payload-pre-processing" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954576 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea87f45c-22b3-4eb0-a1f9-47bc7c795155" containerName="main" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954584 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4258b1b8-9638-4a7f-973c-8209d717bfd0" containerName="istio-proxy" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954594 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c695c6-5acc-481e-a4b0-06c5fe2d933b" containerName="maas-api" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954602 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="874dede1-2042-4f22-8415-4e9497e0b26b" containerName="istio-proxy" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954612 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6b9d59-c7f6-4f43-b2b7-85137885ec3a" containerName="payload-processing" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954622 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f0d23b-ef8c-4f01-a66d-622d13eccd48" containerName="payload-pre-processing" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954632 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bead4e2-f184-49b9-8af8-36c25b7bc7dc" containerName="maas-api" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954639 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954647 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc2c0ac-28c8-4708-b02e-eb77d97fbe77" containerName="payload-processing" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954655 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a181a92-b332-4b5a-bc6a-c8208b899b9c" containerName="payload-processing" Nov 02 14:54:36.955963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.954662 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe3bb4e-5a5c-4019-aada-a451a33c84c6" containerName="payload-processing" Nov 02 14:54:36.959253 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.959232 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:36.976694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.976668 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f30173-openshift-default-dockercfg-tw6wb\"" Nov 02 14:54:36.991528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:36.991495 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng"] Nov 02 14:54:37.046763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.046726 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.046935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.046772 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.046935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.046847 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.046935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.046893 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw222\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-kube-api-access-xw222\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.046935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.046919 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.047107 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.047051 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.047145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.047115 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.047214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.047196 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.047255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.047229 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.148601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.148557 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.148820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.148628 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.148820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.148653 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.148820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.148714 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.148820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.148740 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.148820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.148765 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.149207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.148909 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw222\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-kube-api-access-xw222\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.149207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.148964 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.149207 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.149026 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.149356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.149265 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.149420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.149347 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.149420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.149396 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.149533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.149462 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.149628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.149605 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.151211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.151183 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.151331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.151289 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.173197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.173169 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.173308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.173236 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw222\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-kube-api-access-xw222\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.273531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.273440 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:37.592865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.592834 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng"] Nov 02 14:54:37.594740 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:54:37.594706 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db9246b_adcb_47c3_a49b_d2cadabf8ca7.slice/crio-63a2b68979aee1caa823740897c7fd7bbbba1fc0f7a8cb862b096252ca155cdc WatchSource:0}: Error finding container 63a2b68979aee1caa823740897c7fd7bbbba1fc0f7a8cb862b096252ca155cdc: Status 404 returned error can't find the container with id 63a2b68979aee1caa823740897c7fd7bbbba1fc0f7a8cb862b096252ca155cdc Nov 02 14:54:37.597295 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.597260 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:54:37.597383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.597326 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:54:37.597383 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:37.597355 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:54:38.066168 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.066117 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" event={"ID":"5db9246b-adcb-47c3-a49b-d2cadabf8ca7","Type":"ContainerStarted","Data":"7f9960c655257e33da905e09ba573258d79e6711aba9fc66abccf167e3d7716b"} Nov 02 14:54:38.066470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.066448 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" event={"ID":"5db9246b-adcb-47c3-a49b-d2cadabf8ca7","Type":"ContainerStarted","Data":"63a2b68979aee1caa823740897c7fd7bbbba1fc0f7a8cb862b096252ca155cdc"} Nov 02 14:54:38.124482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.124429 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" podStartSLOduration=2.124414905 podStartE2EDuration="2.124414905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:38.122417698 +0000 UTC m=+4226.848039471" watchObservedRunningTime="2026-11-02 14:54:38.124414905 +0000 UTC m=+4226.850036744" Nov 02 14:54:38.262588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.262551 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:38.282340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.279438 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:38.282855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.282809 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 14:54:38.283150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.283104 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 14:54:38.490391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.490358 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:38.689792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:38.689755 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:39.139055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:39.135836 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:39.274450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:39.274413 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 14:54:39.274641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:39.274468 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 14:54:39.521354 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:39.521263 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:40.274414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:40.274370 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 14:54:40.274855 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:40.274434 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 14:54:40.393816 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:40.393762 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:41.274032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:41.273968 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 14:54:41.274206 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:41.274052 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 14:54:41.714099 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:41.714064 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:41.886087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:41.886049 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:42.274437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:42.274396 5591 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 02 14:54:42.274627 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:42.274453 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 02 14:54:42.378889 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:42.378835 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:42.551651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:42.551520 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:42.721479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:42.721437 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng"] Nov 02 14:54:42.721946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:42.721703 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" containerID="cri-o://7f9960c655257e33da905e09ba573258d79e6711aba9fc66abccf167e3d7716b" gracePeriod=30 Nov 02 14:54:42.805012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:42.804891 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:44.370504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:44.370465 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:46.124860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:46.124822 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:47.588585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:47.588551 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:47.766796 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:47.766735 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:49.457764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:49.457717 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:49.728326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:49.728243 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:54:50.161153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.161118 5591 generic.go:356] "Generic (PLEG): container finished" podID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerID="7f9960c655257e33da905e09ba573258d79e6711aba9fc66abccf167e3d7716b" exitCode=0 Nov 02 14:54:50.161323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.161158 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" event={"ID":"5db9246b-adcb-47c3-a49b-d2cadabf8ca7","Type":"ContainerDied","Data":"7f9960c655257e33da905e09ba573258d79e6711aba9fc66abccf167e3d7716b"} Nov 02 14:54:50.189118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.189090 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:50.384812 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.384779 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-socket\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.384817 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-token\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.384861 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-kube-api-access-xw222\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-kube-api-access-xw222\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.384948 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-envoy\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385042 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istiod-ca-cert\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385122 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-certs\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385163 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-podinfo\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385174 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-socket" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:50.385331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385203 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-credential-socket\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385307 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-data\") pod \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\" (UID: \"5db9246b-adcb-47c3-a49b-d2cadabf8ca7\") " Nov 02 14:54:50.385625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385422 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-certs" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:50.385625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385483 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istiod-ca-cert" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:54:50.385742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385625 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-credential-socket" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:50.385797 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.385779 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-data" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:50.386019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.386002 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:50.386096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.386023 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:50.386096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.386037 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:50.386096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.386059 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:50.386096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.386073 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:50.387373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.387350 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-envoy" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:54:50.387581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.387562 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-token" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:50.387909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.387879 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-kube-api-access-xw222" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "kube-api-access-xw222". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:54:50.388043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.387882 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-podinfo" pod "5db9246b-adcb-47c3-a49b-d2cadabf8ca7" (UID: "5db9246b-adcb-47c3-a49b-d2cadabf8ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:54:50.491800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.491760 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:50.491800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.491796 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:50.491800 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.491808 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:50.492298 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:50.491821 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw222\" (UniqueName: \"kubernetes.io/projected/5db9246b-adcb-47c3-a49b-d2cadabf8ca7-kube-api-access-xw222\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:54:51.136598 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:51.136443 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:51.168577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:51.168541 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" Nov 02 14:54:51.168779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:51.168546 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng" event={"ID":"5db9246b-adcb-47c3-a49b-d2cadabf8ca7","Type":"ContainerDied","Data":"63a2b68979aee1caa823740897c7fd7bbbba1fc0f7a8cb862b096252ca155cdc"} Nov 02 14:54:51.168779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:51.168689 5591 scope.go:122] "RemoveContainer" containerID="7f9960c655257e33da905e09ba573258d79e6711aba9fc66abccf167e3d7716b" Nov 02 14:54:51.220868 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:51.220834 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng"] Nov 02 14:54:51.233041 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:51.233009 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-7hqng"] Nov 02 14:54:51.430907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:51.430824 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" path="/var/lib/kubelet/pods/5db9246b-adcb-47c3-a49b-d2cadabf8ca7/volumes" Nov 02 14:54:52.949123 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:52.948852 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:54.680529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:54.680482 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:56.305641 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:56.305607 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:57.966200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:57.966161 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:54:59.661232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:54:59.661192 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:01.073737 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:55:01.073697 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:55:01.353649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:01.350549 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:03.029939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:03.029576 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:04.655007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:04.651307 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:06.201109 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:06.201056 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:07.871791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:07.871736 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:09.562563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:09.562519 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:11.168905 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:11.166161 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:12.946024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:12.945397 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:14.658786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:14.658686 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:16.358686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:16.358649 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:18.067010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:18.062089 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:19.631009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:19.630303 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:21.348263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:21.346222 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:23.099422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:23.099378 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:24.760380 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:24.760332 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:26.395956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:26.395919 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:28.006677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:28.006628 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:28.572064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:28.572024 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:29.661189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:29.657029 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:31.074774 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:55:31.074733 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:55:31.470831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:31.470781 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:32.151289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:32.151226 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:33.148872 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:33.148823 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:34.923419 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:34.923376 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:36.464391 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:36.464102 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:38.063895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:38.063057 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:38.666551 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:38.666504 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:55:39.654097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:39.654056 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:41.468014 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:41.458822 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:43.174070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:43.173947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:44.762823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:44.762782 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:46.476947 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:46.476851 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:48.172106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:48.172061 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:49.755492 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:49.755438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:51.453129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:51.451317 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:53.044006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:53.043690 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:54.601224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:54.601178 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:56.119968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.119925 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2"] Nov 02 14:55:56.120585 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.120563 5591 cpu_manager.go:414] "containerMap: removing container" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" Nov 02 14:55:56.120669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.120588 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8845a851-daf1-4d64-9341-ea6b4d256ce5" containerName="istio-proxy" Nov 02 14:55:56.120669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.120609 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" Nov 02 14:55:56.120669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.120617 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" Nov 02 14:55:56.120669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.120630 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" Nov 02 14:55:56.120669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.120638 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3317507b-9f40-45b7-a3dc-9bfd1fc20c9c" containerName="istio-proxy" Nov 02 14:55:56.120958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.120797 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db9246b-adcb-47c3-a49b-d2cadabf8ca7" containerName="istio-proxy" Nov 02 14:55:56.123787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.123768 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.150473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.150426 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-0d1ca7-openshift-default-dockercfg-9n4nr\"" Nov 02 14:55:56.153458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.153429 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2"] Nov 02 14:55:56.215768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.215715 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:56.240013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.235891 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6nkn\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-kube-api-access-b6nkn\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.247411 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.247358 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ee53c2-ff7a-496e-be14-6e2e938be315-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.247909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.247820 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.248250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.248228 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.248406 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.248390 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.248596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.248580 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.248756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.248734 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.248919 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.248896 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.249072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.249056 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350373 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350420 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350462 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350491 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350535 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350570 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350640 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6nkn\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-kube-api-access-b6nkn\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350668 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ee53c2-ff7a-496e-be14-6e2e938be315-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.350724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350700 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.351088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350861 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.351088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.350861 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.351214 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.351186 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.351276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.351250 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.351637 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.351612 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ee53c2-ff7a-496e-be14-6e2e938be315-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.353098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.353064 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.353228 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.353213 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.373940 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.373880 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6nkn\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-kube-api-access-b6nkn\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.374457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.374435 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.437737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.437704 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:56.757964 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.757931 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2"] Nov 02 14:55:56.760269 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:55:56.760229 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ee53c2_ff7a_496e_be14_6e2e938be315.slice/crio-54c330021615271ab389d8d4a0c14c7c30413e449e69158c5ecb1d148f8c94b0 WatchSource:0}: Error finding container 54c330021615271ab389d8d4a0c14c7c30413e449e69158c5ecb1d148f8c94b0: Status 404 returned error can't find the container with id 54c330021615271ab389d8d4a0c14c7c30413e449e69158c5ecb1d148f8c94b0 Nov 02 14:55:56.762833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.762797 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:55:56.762958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.762879 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:55:56.762958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:56.762925 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:55:57.659516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:57.659470 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" event={"ID":"62ee53c2-ff7a-496e-be14-6e2e938be315","Type":"ContainerStarted","Data":"82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c"} Nov 02 14:55:57.660071 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:57.659539 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" event={"ID":"62ee53c2-ff7a-496e-be14-6e2e938be315","Type":"ContainerStarted","Data":"54c330021615271ab389d8d4a0c14c7c30413e449e69158c5ecb1d148f8c94b0"} Nov 02 14:55:57.718073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:57.718006 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" podStartSLOduration=1.717971044 podStartE2EDuration="1.717971044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:57.71641659 +0000 UTC m=+4306.442038360" watchObservedRunningTime="2026-11-02 14:55:57.717971044 +0000 UTC m=+4306.443592813" Nov 02 14:55:57.957058 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:57.953737 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:55:58.456160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:58.456119 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:55:58.458063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:58.458033 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 14:55:58.458250 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:58.458093 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 14:55:59.457008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:59.444785 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 14:55:59.457008 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:59.444855 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 14:55:59.895457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:55:59.895422 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:00.452009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:00.450540 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 14:56:00.452009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:00.450611 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 14:56:01.081745 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:01.081709 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:56:01.442004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:01.439548 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 14:56:01.442004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:01.439616 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 14:56:01.562010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:01.561891 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:02.454010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:02.448899 5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body= Nov 02 14:56:02.454010 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:02.448961 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" Nov 02 14:56:03.234054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:03.233522 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:03.597437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:03.597317 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:56:03.598213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:03.598174 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:56:03.599481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:03.599452 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:56:04.321865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.321828 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7033-573cc9"] Nov 02 14:56:04.326154 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.326133 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:04.342044 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.342015 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7033-573cc9"] Nov 02 14:56:04.351340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.351316 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7jzh4\"" Nov 02 14:56:04.520996 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.520952 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpq8\" (UniqueName: \"kubernetes.io/projected/d8dec5b0-8891-47fc-af44-c52494a112a3-kube-api-access-bfpq8\") pod \"test-curl-7033-573cc9\" (UID: \"d8dec5b0-8891-47fc-af44-c52494a112a3\") " pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:04.622682 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.622603 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpq8\" (UniqueName: \"kubernetes.io/projected/d8dec5b0-8891-47fc-af44-c52494a112a3-kube-api-access-bfpq8\") pod \"test-curl-7033-573cc9\" (UID: \"d8dec5b0-8891-47fc-af44-c52494a112a3\") " pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:04.646108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.646074 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpq8\" (UniqueName: \"kubernetes.io/projected/d8dec5b0-8891-47fc-af44-c52494a112a3-kube-api-access-bfpq8\") pod \"test-curl-7033-573cc9\" (UID: \"d8dec5b0-8891-47fc-af44-c52494a112a3\") " pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:04.919132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.919094 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:04.938826 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:04.938787 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:05.246193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:05.246156 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7033-573cc9"] Nov 02 14:56:05.248442 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:56:05.248415 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8dec5b0_8891_47fc_af44_c52494a112a3.slice/crio-585026b70cd36c456f321ed7a6578c958f1e64e1e6a1146d8b4e5c9a6cbe9fad WatchSource:0}: Error finding container 585026b70cd36c456f321ed7a6578c958f1e64e1e6a1146d8b4e5c9a6cbe9fad: Status 404 returned error can't find the container with id 585026b70cd36c456f321ed7a6578c958f1e64e1e6a1146d8b4e5c9a6cbe9fad Nov 02 14:56:05.710975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:05.710931 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7033-573cc9" event={"ID":"d8dec5b0-8891-47fc-af44-c52494a112a3","Type":"ContainerStarted","Data":"936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0"} Nov 02 14:56:05.710975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:05.710995 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7033-573cc9" event={"ID":"d8dec5b0-8891-47fc-af44-c52494a112a3","Type":"ContainerStarted","Data":"585026b70cd36c456f321ed7a6578c958f1e64e1e6a1146d8b4e5c9a6cbe9fad"} Nov 02 14:56:05.750060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:05.750007 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7033-573cc9" podStartSLOduration=1.7499788010000001 podStartE2EDuration="1.749978801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:05.748082351 +0000 UTC m=+4314.473704121" watchObservedRunningTime="2026-11-02 14:56:05.749978801 +0000 UTC m=+4314.475600570" Nov 02 14:56:06.623155 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.621295 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:06.633082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.633046 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7033-573cc9"] Nov 02 14:56:06.635181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.635151 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7033-573cc9"] Nov 02 14:56:06.754930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.754891 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7033-ee22c5"] Nov 02 14:56:06.755693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.755668 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d8dec5b0-8891-47fc-af44-c52494a112a3" containerName="test-curl-7033-573cc9" Nov 02 14:56:06.755794 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.755694 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dec5b0-8891-47fc-af44-c52494a112a3" containerName="test-curl-7033-573cc9" Nov 02 14:56:06.755853 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.755835 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dec5b0-8891-47fc-af44-c52494a112a3" containerName="test-curl-7033-573cc9" Nov 02 14:56:06.759323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.759298 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7033-ee22c5" Nov 02 14:56:06.775533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.775501 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7033-ee22c5"] Nov 02 14:56:06.850588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.850556 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxl2\" (UniqueName: \"kubernetes.io/projected/40992517-7563-4c5a-8395-8f560b3042ec-kube-api-access-xmxl2\") pod \"test-curl-7033-ee22c5\" (UID: \"40992517-7563-4c5a-8395-8f560b3042ec\") " pod="opendatahub/test-curl-7033-ee22c5" Nov 02 14:56:06.952127 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.952084 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxl2\" (UniqueName: \"kubernetes.io/projected/40992517-7563-4c5a-8395-8f560b3042ec-kube-api-access-xmxl2\") pod \"test-curl-7033-ee22c5\" (UID: \"40992517-7563-4c5a-8395-8f560b3042ec\") " pod="opendatahub/test-curl-7033-ee22c5" Nov 02 14:56:06.975781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:06.975750 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxl2\" (UniqueName: \"kubernetes.io/projected/40992517-7563-4c5a-8395-8f560b3042ec-kube-api-access-xmxl2\") pod \"test-curl-7033-ee22c5\" (UID: \"40992517-7563-4c5a-8395-8f560b3042ec\") " pod="opendatahub/test-curl-7033-ee22c5" Nov 02 14:56:07.072330 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:07.072290 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7033-ee22c5" Nov 02 14:56:07.185874 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:07.185814 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:07.386977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:07.386944 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7033-ee22c5"] Nov 02 14:56:07.388998 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:56:07.388954 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40992517_7563_4c5a_8395_8f560b3042ec.slice/crio-f302af8586377d4a65bc53703cb7d05af641d598090f09a0edf80859b303848b WatchSource:0}: Error finding container f302af8586377d4a65bc53703cb7d05af641d598090f09a0edf80859b303848b: Status 404 returned error can't find the container with id f302af8586377d4a65bc53703cb7d05af641d598090f09a0edf80859b303848b Nov 02 14:56:07.723972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:07.723868 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7033-ee22c5" event={"ID":"40992517-7563-4c5a-8395-8f560b3042ec","Type":"ContainerStarted","Data":"281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645"} Nov 02 14:56:07.723972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:07.723901 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7033-573cc9" podUID="d8dec5b0-8891-47fc-af44-c52494a112a3" containerName="test-curl-7033-573cc9" containerID="cri-o://936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0" gracePeriod=2 Nov 02 14:56:07.723972 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:07.723923 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7033-ee22c5" event={"ID":"40992517-7563-4c5a-8395-8f560b3042ec","Type":"ContainerStarted","Data":"f302af8586377d4a65bc53703cb7d05af641d598090f09a0edf80859b303848b"} Nov 02 14:56:07.762953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:07.762899 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7033-ee22c5" podStartSLOduration=1.762885705 podStartE2EDuration="1.762885705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:07.761161337 +0000 UTC m=+4316.486783104" watchObservedRunningTime="2026-11-02 14:56:07.762885705 +0000 UTC m=+4316.488507474" Nov 02 14:56:08.275544 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:08.275508 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:08.617028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:08.616929 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7033-ee22c5"] Nov 02 14:56:08.624252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:08.624220 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7033-ee22c5"] Nov 02 14:56:09.735375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:09.735337 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7033-ee22c5" podUID="40992517-7563-4c5a-8395-8f560b3042ec" containerName="test-curl-7033-ee22c5" containerID="cri-o://281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645" gracePeriod=2 Nov 02 14:56:09.933728 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:09.933698 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:09.987482 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:09.987391 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:10.002116 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:10.002082 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8dec5b0-8891-47fc-af44-c52494a112a3-kube-api-access-bfpq8\" (UniqueName: \"kubernetes.io/projected/d8dec5b0-8891-47fc-af44-c52494a112a3-kube-api-access-bfpq8\") pod \"d8dec5b0-8891-47fc-af44-c52494a112a3\" (UID: \"d8dec5b0-8891-47fc-af44-c52494a112a3\") " Nov 02 14:56:10.004213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:10.004185 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dec5b0-8891-47fc-af44-c52494a112a3-kube-api-access-bfpq8" pod "d8dec5b0-8891-47fc-af44-c52494a112a3" (UID: "d8dec5b0-8891-47fc-af44-c52494a112a3"). InnerVolumeSpecName "kube-api-access-bfpq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:56:10.103884 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:10.103847 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpq8\" (UniqueName: \"kubernetes.io/projected/d8dec5b0-8891-47fc-af44-c52494a112a3-kube-api-access-bfpq8\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:10.742458 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:10.742420 5591 generic.go:356] "Generic (PLEG): container finished" podID="d8dec5b0-8891-47fc-af44-c52494a112a3" containerID="936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0" exitCode=137 Nov 02 14:56:10.742881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:10.742464 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:10.742881 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:10.742509 5591 scope.go:122] "RemoveContainer" containerID="936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0" Nov 02 14:56:10.751829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:10.751811 5591 scope.go:122] "RemoveContainer" containerID="936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0" Nov 02 14:56:10.752129 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:10.752106 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0\": container with ID starting with 936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0 not found: ID does not exist" containerID="936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0" Nov 02 14:56:10.752194 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:10.752138 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0"} err="failed to get container status \"936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0\": rpc error: code = NotFound desc = could not find container \"936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0\": container with ID starting with 936e314aa749ab3a7bddffa512efd7c100d1d4678af85007a55e30255eefc5f0 not found: ID does not exist" Nov 02 14:56:10.764184 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:10.764154 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7033-573cc9\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="d8dec5b0-8891-47fc-af44-c52494a112a3" pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:11.432685 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:11.432650 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8dec5b0-8891-47fc-af44-c52494a112a3" path="/var/lib/kubelet/pods/d8dec5b0-8891-47fc-af44-c52494a112a3/volumes" Nov 02 14:56:11.441848 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:11.441801 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7033-573cc9\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="d8dec5b0-8891-47fc-af44-c52494a112a3" pod="opendatahub/test-curl-7033-573cc9" Nov 02 14:56:11.748581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:11.748550 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:11.954343 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:11.954314 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7033-ee22c5" Nov 02 14:56:12.027695 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:12.027609 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40992517-7563-4c5a-8395-8f560b3042ec-kube-api-access-xmxl2\" (UniqueName: \"kubernetes.io/projected/40992517-7563-4c5a-8395-8f560b3042ec-kube-api-access-xmxl2\") pod \"40992517-7563-4c5a-8395-8f560b3042ec\" (UID: \"40992517-7563-4c5a-8395-8f560b3042ec\") " Nov 02 14:56:12.029692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:12.029668 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40992517-7563-4c5a-8395-8f560b3042ec-kube-api-access-xmxl2" pod "40992517-7563-4c5a-8395-8f560b3042ec" (UID: "40992517-7563-4c5a-8395-8f560b3042ec"). InnerVolumeSpecName "kube-api-access-xmxl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:56:12.129916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:12.129876 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxl2\" (UniqueName: \"kubernetes.io/projected/40992517-7563-4c5a-8395-8f560b3042ec-kube-api-access-xmxl2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:12.757737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:12.757697 5591 generic.go:356] "Generic (PLEG): container finished" podID="40992517-7563-4c5a-8395-8f560b3042ec" containerID="281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645" exitCode=137 Nov 02 14:56:12.758182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:12.757740 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7033-ee22c5" Nov 02 14:56:12.758182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:12.757753 5591 scope.go:122] "RemoveContainer" containerID="281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645" Nov 02 14:56:12.766564 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:12.766547 5591 scope.go:122] "RemoveContainer" containerID="281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645" Nov 02 14:56:12.766837 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:12.766818 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645\": container with ID starting with 281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645 not found: ID does not exist" containerID="281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645" Nov 02 14:56:12.766912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:12.766844 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645"} err="failed to get container status \"281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645\": rpc error: code = NotFound desc = could not find container \"281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645\": container with ID starting with 281f22a3e79fe2145a3b7de5ac3ae939781d2073be286035f80009f7a8cd5645 not found: ID does not exist" Nov 02 14:56:12.779556 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:12.779525 5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7033-ee22c5\" is forbidden: User \"system:node:ip-10-0-140-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-236.ec2.internal' and this object" podUID="40992517-7563-4c5a-8395-8f560b3042ec" pod="opendatahub/test-curl-7033-ee22c5" Nov 02 14:56:13.408702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:13.408651 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:13.518255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:13.518179 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40992517-7563-4c5a-8395-8f560b3042ec" path="/var/lib/kubelet/pods/40992517-7563-4c5a-8395-8f560b3042ec/volumes" Nov 02 14:56:13.594902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:13.594866 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:14.429561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:14.429524 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:56:15.065469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:15.065357 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:15.667004 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:15.666863 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:16.083493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:16.083400 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:16.645896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:16.645583 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:17.859786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:17.857949 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:18.287236 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:18.287196 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:18.472366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:18.472329 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:19.986137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:19.986097 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:21.461363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:21.461329 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7"] Nov 02 14:56:21.680963 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:21.680928 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:21.882422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:21.882381 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:23.565527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:23.565446 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:23.733890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:23.733857 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:25.099876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:25.099836 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:25.258978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:25.258939 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:25.420457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:25.420412 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:26.461803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:26.461764 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" podUID="13100993-e396-430e-9692-694b284426fc" containerName="payload-processing" containerID="cri-o://59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512" gracePeriod=25 Nov 02 14:56:26.730328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:26.730210 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:27.352582 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:27.352527 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:28.696900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:28.696865 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:29.181344 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:29.181303 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:30.779673 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:30.779637 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:30.941911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:30.941878 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:31.073912 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:31.073827 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:56:31.668182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:31.664357 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:31.869915 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:31.869879 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:32.387132 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:32.374940 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:32.714831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:32.714698 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:32.935136 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:32.935099 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" podUID="13100993-e396-430e-9692-694b284426fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:56:32.978187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:32.978091 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:33.758246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:33.758208 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:34.433418 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:34.433381 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:36.150479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:36.150440 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:37.897976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:37.897936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:39.309331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:39.309295 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm"] Nov 02 14:56:40.527509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:40.527473 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:40.696576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:40.696544 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:40.945597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:40.945558 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:41.552834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:41.552794 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:42.935524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:42.935484 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" podUID="13100993-e396-430e-9692-694b284426fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:56:43.249279 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:43.249187 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:44.310103 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:44.310050 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" podUID="bdde9b7a-bec6-4c34-a647-2448a0301ff6" containerName="payload-processing" containerID="cri-o://49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34" gracePeriod=25 Nov 02 14:56:44.548317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:44.548282 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c"] Nov 02 14:56:44.548519 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:44.548493 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" podUID="43b2c8e3-58b7-447b-b179-1e83d6543efe" containerName="maas-api" containerID="cri-o://c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d" gracePeriod=30 Nov 02 14:56:44.662680 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:44.662616 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:44.904159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:44.904125 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:56:45.003005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.002947 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b2c8e3-58b7-447b-b179-1e83d6543efe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b2c8e3-58b7-447b-b179-1e83d6543efe-openshift-service-ca\") pod \"43b2c8e3-58b7-447b-b179-1e83d6543efe\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " Nov 02 14:56:45.003420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.003393 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-metrics-certs\") pod \"43b2c8e3-58b7-447b-b179-1e83d6543efe\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " Nov 02 14:56:45.003600 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.003583 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b2c8e3-58b7-447b-b179-1e83d6543efe-kube-api-access-xd8rx\" (UniqueName: \"kubernetes.io/projected/43b2c8e3-58b7-447b-b179-1e83d6543efe-kube-api-access-xd8rx\") pod \"43b2c8e3-58b7-447b-b179-1e83d6543efe\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " Nov 02 14:56:45.003718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.003704 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-maas-api-tls\") pod \"43b2c8e3-58b7-447b-b179-1e83d6543efe\" (UID: \"43b2c8e3-58b7-447b-b179-1e83d6543efe\") " Nov 02 14:56:45.005645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.005603 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b2c8e3-58b7-447b-b179-1e83d6543efe-openshift-service-ca" pod "43b2c8e3-58b7-447b-b179-1e83d6543efe" (UID: "43b2c8e3-58b7-447b-b179-1e83d6543efe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:56:45.006290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.006266 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b2c8e3-58b7-447b-b179-1e83d6543efe-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:45.088832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.088792 5591 generic.go:356] "Generic (PLEG): container finished" podID="43b2c8e3-58b7-447b-b179-1e83d6543efe" containerID="c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d" exitCode=0 Nov 02 14:56:45.089091 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.089049 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" event={"ID":"43b2c8e3-58b7-447b-b179-1e83d6543efe","Type":"ContainerDied","Data":"c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d"} Nov 02 14:56:45.089182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.089106 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" event={"ID":"43b2c8e3-58b7-447b-b179-1e83d6543efe","Type":"ContainerDied","Data":"5e0be3d2be21f5a8ecaaaa9ea4c6c8e7c2a94914dc5f79fddb1dcffbf4e005ab"} Nov 02 14:56:45.089182 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.089138 5591 scope.go:122] "RemoveContainer" containerID="c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d" Nov 02 14:56:45.089340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.089316 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c" Nov 02 14:56:45.099427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.099393 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-maas-api-tls" pod "43b2c8e3-58b7-447b-b179-1e83d6543efe" (UID: "43b2c8e3-58b7-447b-b179-1e83d6543efe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:56:45.102114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.101550 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-metrics-certs" pod "43b2c8e3-58b7-447b-b179-1e83d6543efe" (UID: "43b2c8e3-58b7-447b-b179-1e83d6543efe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:56:45.102114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.101712 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b2c8e3-58b7-447b-b179-1e83d6543efe-kube-api-access-xd8rx" pod "43b2c8e3-58b7-447b-b179-1e83d6543efe" (UID: "43b2c8e3-58b7-447b-b179-1e83d6543efe"). InnerVolumeSpecName "kube-api-access-xd8rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:56:45.115651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.115619 5591 scope.go:122] "RemoveContainer" containerID="c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d" Nov 02 14:56:45.116055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.116026 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:45.116158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.116063 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd8rx\" (UniqueName: \"kubernetes.io/projected/43b2c8e3-58b7-447b-b179-1e83d6543efe-kube-api-access-xd8rx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:45.116158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.116076 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b2c8e3-58b7-447b-b179-1e83d6543efe-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:45.116158 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:45.116026 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d\": container with ID starting with c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d not found: ID does not exist" containerID="c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d" Nov 02 14:56:45.116158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.116100 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d"} err="failed to get container status \"c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d\": rpc error: code = NotFound desc = could not find container \"c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d\": container with ID starting with c4574c753d22a347e9caa1ba4163f0a9f923881a3f0a3f2064797128cc8d259d not found: ID does not exist" Nov 02 14:56:45.244069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.243569 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:45.436472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.436435 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c"] Nov 02 14:56:45.440208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:45.440183 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-26b508-6dc787c8d9-5p48c"] Nov 02 14:56:46.717480 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.717416 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:56:46.771468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.771436 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:46.835795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.835744 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13100993-e396-430e-9692-694b284426fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13100993-e396-430e-9692-694b284426fc-plugins-config-volume\") pod \"13100993-e396-430e-9692-694b284426fc\" (UID: \"13100993-e396-430e-9692-694b284426fc\") " Nov 02 14:56:46.835795 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.835789 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13100993-e396-430e-9692-694b284426fc-kube-api-access-xbq8w\" (UniqueName: \"kubernetes.io/projected/13100993-e396-430e-9692-694b284426fc-kube-api-access-xbq8w\") pod \"13100993-e396-430e-9692-694b284426fc\" (UID: \"13100993-e396-430e-9692-694b284426fc\") " Nov 02 14:56:46.836204 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.836180 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13100993-e396-430e-9692-694b284426fc-plugins-config-volume" pod "13100993-e396-430e-9692-694b284426fc" (UID: "13100993-e396-430e-9692-694b284426fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:56:46.838189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.838160 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13100993-e396-430e-9692-694b284426fc-kube-api-access-xbq8w" pod "13100993-e396-430e-9692-694b284426fc" (UID: "13100993-e396-430e-9692-694b284426fc"). InnerVolumeSpecName "kube-api-access-xbq8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:56:46.931817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.931774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:46.937321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.937294 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13100993-e396-430e-9692-694b284426fc-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:46.937321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:46.937322 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbq8w\" (UniqueName: \"kubernetes.io/projected/13100993-e396-430e-9692-694b284426fc-kube-api-access-xbq8w\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:47.105057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.104940 5591 generic.go:356] "Generic (PLEG): container finished" podID="13100993-e396-430e-9692-694b284426fc" containerID="59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512" exitCode=0 Nov 02 14:56:47.105057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.105015 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" Nov 02 14:56:47.105057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.105019 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" event={"ID":"13100993-e396-430e-9692-694b284426fc","Type":"ContainerDied","Data":"59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512"} Nov 02 14:56:47.105297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.105074 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7" event={"ID":"13100993-e396-430e-9692-694b284426fc","Type":"ContainerDied","Data":"8df1e27e67217df30099fef1dda032b38363d865958adcfa3f9059639adf49ae"} Nov 02 14:56:47.105297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.105096 5591 scope.go:122] "RemoveContainer" containerID="59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512" Nov 02 14:56:47.114686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.114668 5591 scope.go:122] "RemoveContainer" containerID="59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512" Nov 02 14:56:47.114950 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:47.114932 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512\": container with ID starting with 59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512 not found: ID does not exist" containerID="59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512" Nov 02 14:56:47.115069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.114958 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512"} err="failed to get container status \"59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512\": rpc error: code = NotFound desc = could not find container \"59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512\": container with ID starting with 59e1f5787f2aebf327cec359b670165545f178b85fb1f3f25e173c0fbba0a512 not found: ID does not exist" Nov 02 14:56:47.157694 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.157663 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7"] Nov 02 14:56:47.169449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.169418 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-26b508-55f4ddcb65-kfvn7"] Nov 02 14:56:47.297408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.285290 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:47.431138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.431106 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13100993-e396-430e-9692-694b284426fc" path="/var/lib/kubelet/pods/13100993-e396-430e-9692-694b284426fc/volumes" Nov 02 14:56:47.431604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.431588 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b2c8e3-58b7-447b-b179-1e83d6543efe" path="/var/lib/kubelet/pods/43b2c8e3-58b7-447b-b179-1e83d6543efe/volumes" Nov 02 14:56:47.465208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.465178 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:47.546999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:47.546949 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" podUID="bdde9b7a-bec6-4c34-a647-2448a0301ff6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:56:48.036584 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:48.033500 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:48.320559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:48.320469 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c"] Nov 02 14:56:48.320765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:48.320735 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" containerID="cri-o://4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09" gracePeriod=30 Nov 02 14:56:48.425802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:48.410210 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:48.633163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:48.631654 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:49.190112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:49.190076 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:50.373334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:50.373216 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:51.824935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:51.824898 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk"] Nov 02 14:56:51.911556 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:51.911501 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:52.073030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:52.072951 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:53.585876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.585845 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:56:53.625887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.625823 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-certs\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.625918 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istiod-ca-cert\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.625966 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-podinfo\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626071 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-socket\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626136 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-envoy\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626152 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-certs" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:56:53.626216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626186 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-token\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-token\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626233 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-data\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626266 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-credential-socket\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626318 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-kube-api-access-v8ztm\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-kube-api-access-v8ztm\") pod \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\" (UID: \"c39babc9-1aa2-4878-8b21-b98ddd3a7f34\") " Nov 02 14:56:53.626466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626354 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-socket" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:56:53.626466 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626364 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istiod-ca-cert" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:56:53.626747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626539 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-data" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:56:53.626747 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626637 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-credential-socket" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:56:53.626870 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626851 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.626936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626876 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.626936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626890 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.626936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626904 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.626936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.626920 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.628320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.628299 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-envoy" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:56:53.628438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.628419 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-token" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:56:53.628829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.628807 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-podinfo" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:56:53.628930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.628831 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-kube-api-access-v8ztm" pod "c39babc9-1aa2-4878-8b21-b98ddd3a7f34" (UID: "c39babc9-1aa2-4878-8b21-b98ddd3a7f34"). InnerVolumeSpecName "kube-api-access-v8ztm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:56:53.729361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.729323 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ztm\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-kube-api-access-v8ztm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.729361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.729358 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.729361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.729367 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.729617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.729375 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c39babc9-1aa2-4878-8b21-b98ddd3a7f34-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:56:53.785119 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.785064 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:53.950563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:53.950532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:54.149907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.149861 5591 generic.go:356] "Generic (PLEG): container finished" podID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerID="4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09" exitCode=0 Nov 02 14:56:54.150156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.150052 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" event={"ID":"c39babc9-1aa2-4878-8b21-b98ddd3a7f34","Type":"ContainerDied","Data":"4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09"} Nov 02 14:56:54.150156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.150108 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" event={"ID":"c39babc9-1aa2-4878-8b21-b98ddd3a7f34","Type":"ContainerDied","Data":"dadc441472f5a87639ac04c1accfa9bfc93803fb26ca5f4e123e8caee4d92103"} Nov 02 14:56:54.150156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.150139 5591 scope.go:122] "RemoveContainer" containerID="4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09" Nov 02 14:56:54.150318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.150283 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c" Nov 02 14:56:54.165075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.165048 5591 scope.go:122] "RemoveContainer" containerID="6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407" Nov 02 14:56:54.174709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.174676 5591 scope.go:122] "RemoveContainer" containerID="4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09" Nov 02 14:56:54.175032 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:54.175000 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09\": container with ID starting with 4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09 not found: ID does not exist" containerID="4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09" Nov 02 14:56:54.175140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.175043 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09"} err="failed to get container status \"4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09\": rpc error: code = NotFound desc = could not find container \"4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09\": container with ID starting with 4577ca45d40b83863fd36f93adf6a7b2065114d0687b28e624026e1cba9d7a09 not found: ID does not exist" Nov 02 14:56:54.175140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.175064 5591 scope.go:122] "RemoveContainer" containerID="6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407" Nov 02 14:56:54.175352 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:56:54.175333 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407\": container with ID starting with 6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407 not found: ID does not exist" containerID="6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407" Nov 02 14:56:54.175463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.175364 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407"} err="failed to get container status \"6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407\": rpc error: code = NotFound desc = could not find container \"6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407\": container with ID starting with 6c3f38ee4c746a54a276b0d1e39528f091c45e9c2c9826269c4ce8c649d66407 not found: ID does not exist" Nov 02 14:56:54.206157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.206065 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c"] Nov 02 14:56:54.218294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.216721 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-26b508-openshift-default-767d7d8465-5656c"] Nov 02 14:56:54.300082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:54.300046 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:55.438883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:55.438846 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" path="/var/lib/kubelet/pods/c39babc9-1aa2-4878-8b21-b98ddd3a7f34/volumes" Nov 02 14:56:55.476565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:55.476516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:55.653707 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:55.653671 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:55.813063 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:55.812975 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:55.901854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:55.901815 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:56:57.135753 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:57.135716 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:57.547474 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:57.547378 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" podUID="bdde9b7a-bec6-4c34-a647-2448a0301ff6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:56:57.680467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:57.680431 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:57.867767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:57.867632 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:56:59.556450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:56:59.556411 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:00.708316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:00.708282 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:01.079500 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:01.079398 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:57:01.172397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:01.172365 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:01.332825 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:01.332738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:01.494604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:01.494568 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:01.695421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:01.695367 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:02.907525 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:02.907486 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:03.475701 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:03.475602 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:04.351520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.351467 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:04.512012 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.511946 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:04.573935 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.573872 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:57:04.610247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.610215 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdde9b7a-bec6-4c34-a647-2448a0301ff6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9b7a-bec6-4c34-a647-2448a0301ff6-plugins-config-volume\") pod \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\" (UID: \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\") " Nov 02 14:57:04.610409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.610318 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdde9b7a-bec6-4c34-a647-2448a0301ff6-kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/bdde9b7a-bec6-4c34-a647-2448a0301ff6-kube-api-access-p92v6\") pod \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\" (UID: \"bdde9b7a-bec6-4c34-a647-2448a0301ff6\") " Nov 02 14:57:04.610590 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.610567 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdde9b7a-bec6-4c34-a647-2448a0301ff6-plugins-config-volume" pod "bdde9b7a-bec6-4c34-a647-2448a0301ff6" (UID: "bdde9b7a-bec6-4c34-a647-2448a0301ff6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:04.610823 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.610802 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdde9b7a-bec6-4c34-a647-2448a0301ff6-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:04.612434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.612414 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdde9b7a-bec6-4c34-a647-2448a0301ff6-kube-api-access-p92v6" pod "bdde9b7a-bec6-4c34-a647-2448a0301ff6" (UID: "bdde9b7a-bec6-4c34-a647-2448a0301ff6"). InnerVolumeSpecName "kube-api-access-p92v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:04.712018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:04.711939 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p92v6\" (UniqueName: \"kubernetes.io/projected/bdde9b7a-bec6-4c34-a647-2448a0301ff6-kube-api-access-p92v6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:05.256233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.256197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:05.265815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.265780 5591 generic.go:356] "Generic (PLEG): container finished" podID="bdde9b7a-bec6-4c34-a647-2448a0301ff6" containerID="49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34" exitCode=0 Nov 02 14:57:05.265950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.265823 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" Nov 02 14:57:05.265950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.265865 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" event={"ID":"bdde9b7a-bec6-4c34-a647-2448a0301ff6","Type":"ContainerDied","Data":"49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34"} Nov 02 14:57:05.265950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.265918 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm" event={"ID":"bdde9b7a-bec6-4c34-a647-2448a0301ff6","Type":"ContainerDied","Data":"f1d7ca464cd804aaadb668f9520a0548a80d82585309fcd65852e6d67600fc72"} Nov 02 14:57:05.265950 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.265941 5591 scope.go:122] "RemoveContainer" containerID="49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34" Nov 02 14:57:05.275750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.275729 5591 scope.go:122] "RemoveContainer" containerID="49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34" Nov 02 14:57:05.276079 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:05.276059 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34\": container with ID starting with 49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34 not found: ID does not exist" containerID="49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34" Nov 02 14:57:05.276157 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.276088 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34"} err="failed to get container status \"49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34\": rpc error: code = NotFound desc = could not find container \"49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34\": container with ID starting with 49a5f14319a83f313b95cd2e349854a8263b8a0c2bd0af1f567871d5a8d14f34 not found: ID does not exist" Nov 02 14:57:05.348101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.348060 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm"] Nov 02 14:57:05.349798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.349581 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-66bc6d-57bbb59955-nhlcm"] Nov 02 14:57:05.422911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.422879 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:05.429757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.429728 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdde9b7a-bec6-4c34-a647-2448a0301ff6" path="/var/lib/kubelet/pods/bdde9b7a-bec6-4c34-a647-2448a0301ff6/volumes" Nov 02 14:57:05.934792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:05.934758 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:06.754059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:06.746713 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:06.845461 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:06.845417 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" podUID="39540df0-9900-4ded-8dd2-d21ed94e1dbc" containerName="main" containerID="cri-o://4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c" gracePeriod=45 Nov 02 14:57:06.920029 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:06.919959 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:07.102262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.102176 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:07.147609 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.147579 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:57:07.276077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.275955 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tmp-dir\") pod \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " Nov 02 14:57:07.276077 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276054 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39540df0-9900-4ded-8dd2-d21ed94e1dbc-kube-api-access-bgrz4\" (UniqueName: \"kubernetes.io/projected/39540df0-9900-4ded-8dd2-d21ed94e1dbc-kube-api-access-bgrz4\") pod \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " Nov 02 14:57:07.276307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276105 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-home\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-home\") pod \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " Nov 02 14:57:07.276307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276143 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-model-cache\") pod \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " Nov 02 14:57:07.276307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276259 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tmp-dir" pod "39540df0-9900-4ded-8dd2-d21ed94e1dbc" (UID: "39540df0-9900-4ded-8dd2-d21ed94e1dbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:07.276307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276268 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-dshm\") pod \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " Nov 02 14:57:07.276545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276400 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-home" pod "39540df0-9900-4ded-8dd2-d21ed94e1dbc" (UID: "39540df0-9900-4ded-8dd2-d21ed94e1dbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:07.276545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276412 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-model-cache" pod "39540df0-9900-4ded-8dd2-d21ed94e1dbc" (UID: "39540df0-9900-4ded-8dd2-d21ed94e1dbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:07.276545 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276433 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tls-certs\") pod \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\" (UID: \"39540df0-9900-4ded-8dd2-d21ed94e1dbc\") " Nov 02 14:57:07.276822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276795 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:07.276822 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276818 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:07.277019 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.276830 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:07.278614 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.278584 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tls-certs" pod "39540df0-9900-4ded-8dd2-d21ed94e1dbc" (UID: "39540df0-9900-4ded-8dd2-d21ed94e1dbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:07.278730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.278638 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-dshm" pod "39540df0-9900-4ded-8dd2-d21ed94e1dbc" (UID: "39540df0-9900-4ded-8dd2-d21ed94e1dbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:07.278730 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.278672 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39540df0-9900-4ded-8dd2-d21ed94e1dbc-kube-api-access-bgrz4" pod "39540df0-9900-4ded-8dd2-d21ed94e1dbc" (UID: "39540df0-9900-4ded-8dd2-d21ed94e1dbc"). InnerVolumeSpecName "kube-api-access-bgrz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:07.280069 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.280044 5591 generic.go:356] "Generic (PLEG): container finished" podID="39540df0-9900-4ded-8dd2-d21ed94e1dbc" containerID="4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c" exitCode=0 Nov 02 14:57:07.280173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.280088 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" Nov 02 14:57:07.280173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.280133 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" event={"ID":"39540df0-9900-4ded-8dd2-d21ed94e1dbc","Type":"ContainerDied","Data":"4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c"} Nov 02 14:57:07.280265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.280178 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk" event={"ID":"39540df0-9900-4ded-8dd2-d21ed94e1dbc","Type":"ContainerDied","Data":"76e5ea3e9bcb69159109f5c3ea622bbd4ba285519daaeb46986f390c85def46e"} Nov 02 14:57:07.280265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.280198 5591 scope.go:122] "RemoveContainer" containerID="4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c" Nov 02 14:57:07.301621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.301602 5591 scope.go:122] "RemoveContainer" containerID="4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c" Nov 02 14:57:07.301906 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:07.301888 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c\": container with ID starting with 4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c not found: ID does not exist" containerID="4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c" Nov 02 14:57:07.301961 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.301915 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c"} err="failed to get container status \"4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c\": rpc error: code = NotFound desc = could not find container \"4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c\": container with ID starting with 4dd3973f882e7efb17c3f0c27feaf49111f9dc0a4ccd30c19f6ea254a0ba6b9c not found: ID does not exist" Nov 02 14:57:07.339808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.339749 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk"] Nov 02 14:57:07.343363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.343334 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/sub-test-model-960a68-kserve-648f7c5dc7-vqnzk"] Nov 02 14:57:07.378131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.378101 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39540df0-9900-4ded-8dd2-d21ed94e1dbc-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:07.378131 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.378131 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgrz4\" (UniqueName: \"kubernetes.io/projected/39540df0-9900-4ded-8dd2-d21ed94e1dbc-kube-api-access-bgrz4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:07.378314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.378141 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39540df0-9900-4ded-8dd2-d21ed94e1dbc-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:07.430991 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.430944 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39540df0-9900-4ded-8dd2-d21ed94e1dbc" path="/var/lib/kubelet/pods/39540df0-9900-4ded-8dd2-d21ed94e1dbc/volumes" Nov 02 14:57:07.515604 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.515567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:07.898233 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:07.898200 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:08.890995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:08.890958 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:09.059863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:09.059828 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:09.562402 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:09.562365 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:09.730783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:09.730751 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:10.750493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:10.749933 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:12.354908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:12.354855 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:13.949007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:13.948602 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:14.499876 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:14.499838 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:14.829191 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:14.829157 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9"] Nov 02 14:57:15.019092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:15.018972 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:15.185906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:15.185865 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:15.449642 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:15.449585 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:15.897269 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:15.897229 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:16.196377 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:16.196217 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:16.755662 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:16.755627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:17.697047 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:17.697012 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:17.907022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:17.905038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:18.078478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:18.078354 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:18.373615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:18.373522 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:18.677362 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:18.677324 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb"] Nov 02 14:57:18.677621 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:18.677594 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" containerID="cri-o://5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a" gracePeriod=30 Nov 02 14:57:18.808007 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:18.786896 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:19.582060 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:19.581999 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:19.830080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:19.830034 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" podUID="40d0620c-2792-4b2e-8c86-367582a82e2c" containerName="payload-processing" containerID="cri-o://f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632" gracePeriod=25 Nov 02 14:57:19.854022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:19.853789 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:19.928276 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:19.928239 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:57:19.928450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:19.928313 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:57:21.460397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:21.460356 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:22.506830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:22.506794 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:23.218208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:23.218169 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:23.397709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:23.397666 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:23.963840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:23.963815 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:57:24.035289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035254 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-credential-socket\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.035510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035391 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-kube-api-access-6h7r6\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-kube-api-access-6h7r6\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.035510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035426 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-envoy\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.035510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035446 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-istio-token\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-istio-token\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.035510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035484 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98702400-9887-4120-adea-809091fdf0aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98702400-9887-4120-adea-809091fdf0aa-istio-podinfo\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.035752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035532 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-socket\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.035752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035569 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-certs\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.035752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035608 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98702400-9887-4120-adea-809091fdf0aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98702400-9887-4120-adea-809091fdf0aa-istiod-ca-cert\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.035752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035568 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-credential-socket" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:24.035752 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.035699 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-data\") pod \"98702400-9887-4120-adea-809091fdf0aa\" (UID: \"98702400-9887-4120-adea-809091fdf0aa\") " Nov 02 14:57:24.036268 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.036245 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.036437 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.036266 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-certs" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:24.036555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.036246 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-socket" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:24.036672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.036423 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98702400-9887-4120-adea-809091fdf0aa-istiod-ca-cert" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:24.036672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.036456 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-data" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:24.038689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.038619 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98702400-9887-4120-adea-809091fdf0aa-istio-podinfo" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:57:24.038689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.038660 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-kube-api-access-6h7r6" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "kube-api-access-6h7r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:24.038890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.038773 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-istio-token" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:24.039049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.039029 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-envoy" pod "98702400-9887-4120-adea-809091fdf0aa" (UID: "98702400-9887-4120-adea-809091fdf0aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:24.137531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.137494 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.137531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.137528 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7r6\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-kube-api-access-6h7r6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.137531 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.137538 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.137908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.137547 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98702400-9887-4120-adea-809091fdf0aa-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.137908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.137555 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98702400-9887-4120-adea-809091fdf0aa-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.137908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.137563 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.137908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.137572 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98702400-9887-4120-adea-809091fdf0aa-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.137908 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.137581 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98702400-9887-4120-adea-809091fdf0aa-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:24.379201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.379101 5591 generic.go:356] "Generic (PLEG): container finished" podID="98702400-9887-4120-adea-809091fdf0aa" containerID="5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a" exitCode=0 Nov 02 14:57:24.379201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.379140 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" event={"ID":"98702400-9887-4120-adea-809091fdf0aa","Type":"ContainerDied","Data":"5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a"} Nov 02 14:57:24.379201 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.379153 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" Nov 02 14:57:24.379562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.379228 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb" event={"ID":"98702400-9887-4120-adea-809091fdf0aa","Type":"ContainerDied","Data":"092cbabf3685f3c788be9d64b2fb69ddf835182211172662d93ebf4d3bc043ef"} Nov 02 14:57:24.379562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.379255 5591 scope.go:122] "RemoveContainer" containerID="5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a" Nov 02 14:57:24.390005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.389966 5591 scope.go:122] "RemoveContainer" containerID="27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f" Nov 02 14:57:24.399509 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.399363 5591 scope.go:122] "RemoveContainer" containerID="5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a" Nov 02 14:57:24.399704 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:24.399683 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a\": container with ID starting with 5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a not found: ID does not exist" containerID="5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a" Nov 02 14:57:24.399762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.399716 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a"} err="failed to get container status \"5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a\": rpc error: code = NotFound desc = could not find container \"5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a\": container with ID starting with 5e3a33d4da0be0266faf1c50a0757caee236ed8d3bd9fb69284874acb457734a not found: ID does not exist" Nov 02 14:57:24.399762 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.399733 5591 scope.go:122] "RemoveContainer" containerID="27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f" Nov 02 14:57:24.400089 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:24.400067 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f\": container with ID starting with 27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f not found: ID does not exist" containerID="27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f" Nov 02 14:57:24.400173 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.400099 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f"} err="failed to get container status \"27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f\": rpc error: code = NotFound desc = could not find container \"27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f\": container with ID starting with 27e4a86d32357edbba14563e4a59ad2cb4875ff6577eaaeec0521b5a5cc4c16f not found: ID does not exist" Nov 02 14:57:24.400583 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.400565 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:24.431398 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.431367 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb"] Nov 02 14:57:24.443138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.443115 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ee3eca-openshift-default-854d6687fc-55rmb"] Nov 02 14:57:24.979313 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:24.979273 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:25.007306 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:25.007262 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" podUID="40d0620c-2792-4b2e-8c86-367582a82e2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:57:25.431175 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:25.431137 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98702400-9887-4120-adea-809091fdf0aa" path="/var/lib/kubelet/pods/98702400-9887-4120-adea-809091fdf0aa/volumes" Nov 02 14:57:25.455158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:25.455123 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7"] Nov 02 14:57:25.486275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:25.486243 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh"] Nov 02 14:57:25.653436 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:25.653395 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:25.820968 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:25.820851 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:26.348479 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:26.348402 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:26.517815 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:26.517780 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.172326 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:27.172290 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.356638 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:27.356603 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:57:28.811692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:28.811655 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:28.980445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:28.980408 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:29.622329 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:29.622290 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:29.911189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:29.911151 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq"] Nov 02 14:57:29.911751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:29.911415 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" containerID="cri-o://c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2" gracePeriod=30 Nov 02 14:57:30.023017 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:30.018173 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:30.456172 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:30.456112 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" podUID="ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" containerName="payload-processing" containerID="cri-o://1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1" gracePeriod=25 Nov 02 14:57:30.486755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:30.486702 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" podUID="ba4a5237-6a6a-4017-8107-f011ce4fe99e" containerName="payload-pre-processing" containerID="cri-o://dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687" gracePeriod=25 Nov 02 14:57:30.551292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:30.551256 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:30.750459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:30.750378 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:30.918618 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:30.918583 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:31.073889 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:31.073799 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:57:32.163978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:32.163941 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:32.657732 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:32.657668 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:32.823160 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:32.823119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:33.077035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:33.076938 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" podUID="ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:57:33.087909 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:33.087877 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" podUID="ba4a5237-6a6a-4017-8107-f011ce4fe99e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:57:34.180349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:34.180314 5591 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:57:34.180750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:34.180371 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:57:34.547417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:34.547289 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:35.007427 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.007376 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" podUID="40d0620c-2792-4b2e-8c86-367582a82e2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:57:35.167439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.167412 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:57:35.283189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283097 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-token\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-token\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.283189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283183 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-credential-socket\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283292 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-socket\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283347 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istiod-ca-cert\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283396 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-data\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283436 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-envoy\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283441 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-credential-socket" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283531 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-podinfo\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283531 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-socket" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283569 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-certs\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283669 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-data" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:35.283709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283694 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-kube-api-access-g2w59\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-kube-api-access-g2w59\") pod \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\" (UID: \"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c\") " Nov 02 14:57:35.284272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283776 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istiod-ca-cert" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:35.284272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.283904 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-certs" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:35.284272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.284218 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.284272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.284242 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.284272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.284260 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.284272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.284275 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.284630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.284289 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.285455 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.285426 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-token" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:35.285661 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.285640 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-podinfo" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:57:35.285746 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.285660 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-envoy" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:35.285974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.285957 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-kube-api-access-g2w59" pod "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" (UID: "bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c"). InnerVolumeSpecName "kube-api-access-g2w59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:35.385778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.385740 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.385778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.385771 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.385778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.385780 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.386096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.385788 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2w59\" (UniqueName: \"kubernetes.io/projected/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c-kube-api-access-g2w59\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:35.455949 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.455919 5591 generic.go:356] "Generic (PLEG): container finished" podID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerID="c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2" exitCode=0 Nov 02 14:57:35.456115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.455965 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" Nov 02 14:57:35.456115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.456008 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" event={"ID":"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c","Type":"ContainerDied","Data":"c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2"} Nov 02 14:57:35.456115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.456067 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq" event={"ID":"bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c","Type":"ContainerDied","Data":"717b6d155fdfa73d612d3a25746ecc9c6caac7d028120d742681ed7bcecfe903"} Nov 02 14:57:35.456115 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.456098 5591 scope.go:122] "RemoveContainer" containerID="c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2" Nov 02 14:57:35.465337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.465319 5591 scope.go:122] "RemoveContainer" containerID="f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d" Nov 02 14:57:35.473715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.473696 5591 scope.go:122] "RemoveContainer" containerID="c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2" Nov 02 14:57:35.473995 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:35.473960 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2\": container with ID starting with c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2 not found: ID does not exist" containerID="c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2" Nov 02 14:57:35.474064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.474006 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2"} err="failed to get container status \"c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2\": rpc error: code = NotFound desc = could not find container \"c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2\": container with ID starting with c2ad96af9381ca6bb7fd2433b18cf6e137417bae955cfb7dd46a75a9c6b3e8d2 not found: ID does not exist" Nov 02 14:57:35.474064 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.474023 5591 scope.go:122] "RemoveContainer" containerID="f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d" Nov 02 14:57:35.474260 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:35.474241 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d\": container with ID starting with f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d not found: ID does not exist" containerID="f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d" Nov 02 14:57:35.474312 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.474266 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d"} err="failed to get container status \"f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d\": rpc error: code = NotFound desc = could not find container \"f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d\": container with ID starting with f2ead9f88235603b1f1af36308d7109c4f7fdd87355e84e53a6dd666960b395d not found: ID does not exist" Nov 02 14:57:35.506776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.506749 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq"] Nov 02 14:57:35.518803 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.518758 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-drjtq"] Nov 02 14:57:35.960832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:35.960803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:36.565842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:36.565796 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:37.430414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:37.430380 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" path="/var/lib/kubelet/pods/bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c/volumes" Nov 02 14:57:37.822588 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:37.822493 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:37.997328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:37.997286 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:38.167134 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:38.167095 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:38.443055 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:38.428774 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:38.614578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:38.614536 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:39.009263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:39.009215 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:39.762434 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:39.762399 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:40.014062 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.013917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:40.113006 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.112957 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:57:40.165735 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.165703 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d0620c-2792-4b2e-8c86-367582a82e2c-kube-api-access-f9gqh\" (UniqueName: \"kubernetes.io/projected/40d0620c-2792-4b2e-8c86-367582a82e2c-kube-api-access-f9gqh\") pod \"40d0620c-2792-4b2e-8c86-367582a82e2c\" (UID: \"40d0620c-2792-4b2e-8c86-367582a82e2c\") " Nov 02 14:57:40.165946 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.165754 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d0620c-2792-4b2e-8c86-367582a82e2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d0620c-2792-4b2e-8c86-367582a82e2c-plugins-config-volume\") pod \"40d0620c-2792-4b2e-8c86-367582a82e2c\" (UID: \"40d0620c-2792-4b2e-8c86-367582a82e2c\") " Nov 02 14:57:40.166245 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.166214 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d0620c-2792-4b2e-8c86-367582a82e2c-plugins-config-volume" pod "40d0620c-2792-4b2e-8c86-367582a82e2c" (UID: "40d0620c-2792-4b2e-8c86-367582a82e2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:40.166361 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.166339 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d0620c-2792-4b2e-8c86-367582a82e2c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:40.167956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.167923 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d0620c-2792-4b2e-8c86-367582a82e2c-kube-api-access-f9gqh" pod "40d0620c-2792-4b2e-8c86-367582a82e2c" (UID: "40d0620c-2792-4b2e-8c86-367582a82e2c"). InnerVolumeSpecName "kube-api-access-f9gqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:40.191283 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.191235 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:40.267339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.267256 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9gqh\" (UniqueName: \"kubernetes.io/projected/40d0620c-2792-4b2e-8c86-367582a82e2c-kube-api-access-f9gqh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:40.405457 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.405423 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:40.488505 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.488459 5591 generic.go:356] "Generic (PLEG): container finished" podID="40d0620c-2792-4b2e-8c86-367582a82e2c" containerID="f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632" exitCode=0 Nov 02 14:57:40.488720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.488510 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" Nov 02 14:57:40.488720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.488543 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" event={"ID":"40d0620c-2792-4b2e-8c86-367582a82e2c","Type":"ContainerDied","Data":"f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632"} Nov 02 14:57:40.488720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.488598 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9" event={"ID":"40d0620c-2792-4b2e-8c86-367582a82e2c","Type":"ContainerDied","Data":"b2825b222eec1dbc04593c1d37271694dd2943ad7c81b2a5e00a34d94c575a50"} Nov 02 14:57:40.488720 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.488623 5591 scope.go:122] "RemoveContainer" containerID="f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632" Nov 02 14:57:40.499370 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.499351 5591 scope.go:122] "RemoveContainer" containerID="f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632" Nov 02 14:57:40.499667 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:40.499648 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632\": container with ID starting with f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632 not found: ID does not exist" containerID="f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632" Nov 02 14:57:40.499708 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.499677 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632"} err="failed to get container status \"f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632\": rpc error: code = NotFound desc = could not find container \"f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632\": container with ID starting with f74fc9295e754476a70984a94556586e594a3f5663932e4809571c74a675a632 not found: ID does not exist" Nov 02 14:57:40.554558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.554486 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9"] Nov 02 14:57:40.561616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.561591 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02c35-8d7dc5cb9-m7dt9"] Nov 02 14:57:40.853770 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:40.853687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:41.206454 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:41.206422 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2"] Nov 02 14:57:41.206878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:41.206624 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" containerID="cri-o://cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290" gracePeriod=30 Nov 02 14:57:41.305092 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:41.305038 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:41.431596 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:41.431556 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d0620c-2792-4b2e-8c86-367582a82e2c" path="/var/lib/kubelet/pods/40d0620c-2792-4b2e-8c86-367582a82e2c/volumes" Nov 02 14:57:42.290468 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:42.290435 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:42.637839 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:42.637811 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:43.076643 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:43.076497 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" podUID="ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:57:43.105076 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:43.087651 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" podUID="ba4a5237-6a6a-4017-8107-f011ce4fe99e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:57:43.595684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:43.595640 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c"] Nov 02 14:57:43.596246 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:43.595885 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" podUID="afa63769-17d5-4169-96f0-f586c739a12e" containerName="maas-api" containerID="cri-o://56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e" gracePeriod=30 Nov 02 14:57:43.729871 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:43.729840 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:43.898764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:43.898735 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:57:44.005639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.005604 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:44.022156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.022123 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-maas-api-tls\") pod \"afa63769-17d5-4169-96f0-f586c739a12e\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " Nov 02 14:57:44.022386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.022265 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-metrics-certs\") pod \"afa63769-17d5-4169-96f0-f586c739a12e\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " Nov 02 14:57:44.022386 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.022358 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa63769-17d5-4169-96f0-f586c739a12e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa63769-17d5-4169-96f0-f586c739a12e-openshift-service-ca\") pod \"afa63769-17d5-4169-96f0-f586c739a12e\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " Nov 02 14:57:44.022524 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.022404 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa63769-17d5-4169-96f0-f586c739a12e-kube-api-access-zj2w2\" (UniqueName: \"kubernetes.io/projected/afa63769-17d5-4169-96f0-f586c739a12e-kube-api-access-zj2w2\") pod \"afa63769-17d5-4169-96f0-f586c739a12e\" (UID: \"afa63769-17d5-4169-96f0-f586c739a12e\") " Nov 02 14:57:44.022962 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.022939 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa63769-17d5-4169-96f0-f586c739a12e-openshift-service-ca" pod "afa63769-17d5-4169-96f0-f586c739a12e" (UID: "afa63769-17d5-4169-96f0-f586c739a12e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:44.024579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.024552 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-maas-api-tls" pod "afa63769-17d5-4169-96f0-f586c739a12e" (UID: "afa63769-17d5-4169-96f0-f586c739a12e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:44.024700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.024631 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-metrics-certs" pod "afa63769-17d5-4169-96f0-f586c739a12e" (UID: "afa63769-17d5-4169-96f0-f586c739a12e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:44.024766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.024746 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa63769-17d5-4169-96f0-f586c739a12e-kube-api-access-zj2w2" pod "afa63769-17d5-4169-96f0-f586c739a12e" (UID: "afa63769-17d5-4169-96f0-f586c739a12e"). InnerVolumeSpecName "kube-api-access-zj2w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:44.124811 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.124725 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa63769-17d5-4169-96f0-f586c739a12e-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:44.125075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.125053 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2w2\" (UniqueName: \"kubernetes.io/projected/afa63769-17d5-4169-96f0-f586c739a12e-kube-api-access-zj2w2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:44.125304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.125280 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:44.125415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.125404 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa63769-17d5-4169-96f0-f586c739a12e-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:44.368015 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.367818 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:44.510785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.510746 5591 generic.go:356] "Generic (PLEG): container finished" podID="afa63769-17d5-4169-96f0-f586c739a12e" containerID="56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e" exitCode=0 Nov 02 14:57:44.511013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.510800 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" event={"ID":"afa63769-17d5-4169-96f0-f586c739a12e","Type":"ContainerDied","Data":"56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e"} Nov 02 14:57:44.511013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.510815 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" Nov 02 14:57:44.511013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.510842 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c" event={"ID":"afa63769-17d5-4169-96f0-f586c739a12e","Type":"ContainerDied","Data":"d66a72212cc1cd6ffc933105c9ef783f0175d61a25ace129e9f580f0ba03c48a"} Nov 02 14:57:44.511013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.510862 5591 scope.go:122] "RemoveContainer" containerID="56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e" Nov 02 14:57:44.521974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.521955 5591 scope.go:122] "RemoveContainer" containerID="56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e" Nov 02 14:57:44.522253 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:44.522232 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e\": container with ID starting with 56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e not found: ID does not exist" containerID="56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e" Nov 02 14:57:44.522320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.522266 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e"} err="failed to get container status \"56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e\": rpc error: code = NotFound desc = could not find container \"56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e\": container with ID starting with 56fa2b6a3094462e7ab1a3e0b0feaca8bbb9cedf35adb9e3d1c4fb1da32cd41e not found: ID does not exist" Nov 02 14:57:44.565111 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.565073 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c"] Nov 02 14:57:44.575969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:44.575937 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-960a68-99dd6bc8d-kpg2c"] Nov 02 14:57:45.001061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:45.001023 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd"] Nov 02 14:57:45.198327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:45.198289 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:45.381647 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:45.381616 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:45.429382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:45.429351 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa63769-17d5-4169-96f0-f586c739a12e" path="/var/lib/kubelet/pods/afa63769-17d5-4169-96f0-f586c739a12e/volumes" Nov 02 14:57:46.278200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.278153 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:46.469651 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.469619 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:57:46.524558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.524478 5591 generic.go:356] "Generic (PLEG): container finished" podID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerID="cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290" exitCode=0 Nov 02 14:57:46.524558 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.524526 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" Nov 02 14:57:46.524784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.524552 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" event={"ID":"a03005eb-15b1-4d0b-a99b-81e0f207be63","Type":"ContainerDied","Data":"cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290"} Nov 02 14:57:46.524784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.524620 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2" event={"ID":"a03005eb-15b1-4d0b-a99b-81e0f207be63","Type":"ContainerDied","Data":"e56102d39beea8915020ead9385276f0f2bf1a58062c2f27d055620dc2b1483c"} Nov 02 14:57:46.524784 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.524653 5591 scope.go:122] "RemoveContainer" containerID="cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290" Nov 02 14:57:46.536147 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.536126 5591 scope.go:122] "RemoveContainer" containerID="cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290" Nov 02 14:57:46.536463 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:46.536434 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290\": container with ID starting with cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290 not found: ID does not exist" containerID="cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290" Nov 02 14:57:46.536547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.536477 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290"} err="failed to get container status \"cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290\": rpc error: code = NotFound desc = could not find container \"cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290\": container with ID starting with cb496395659bef0b36a76968e851edcdf8447cfad28ee38105ac3ea7307dd290 not found: ID does not exist" Nov 02 14:57:46.577039 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577012 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-kube-api-access-gx9wt\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-kube-api-access-gx9wt\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577060 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03005eb-15b1-4d0b-a99b-81e0f207be63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03005eb-15b1-4d0b-a99b-81e0f207be63-istiod-ca-cert\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577116 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-certs\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577143 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-token\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-token\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577166 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-envoy\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577207 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-credential-socket\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577246 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-data\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577303 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-podinfo\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577397 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-socket\") pod \"a03005eb-15b1-4d0b-a99b-81e0f207be63\" (UID: \"a03005eb-15b1-4d0b-a99b-81e0f207be63\") " Nov 02 14:57:46.577671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577470 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-certs" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:46.577671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577592 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-credential-socket" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:46.577671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577608 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-data" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:46.577671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577645 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03005eb-15b1-4d0b-a99b-81e0f207be63-istiod-ca-cert" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:46.577887 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577785 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-socket" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:46.577958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577940 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.578030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577960 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.578030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.577976 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03005eb-15b1-4d0b-a99b-81e0f207be63-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.578030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.578019 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.578211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.578031 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.579615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.579587 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-envoy" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:46.579744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.579683 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-kube-api-access-gx9wt" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "kube-api-access-gx9wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:46.579894 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.579877 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-token" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:46.580037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.579978 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-podinfo" pod "a03005eb-15b1-4d0b-a99b-81e0f207be63" (UID: "a03005eb-15b1-4d0b-a99b-81e0f207be63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:57:46.679059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.679025 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.679059 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.679056 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.679275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.679068 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03005eb-15b1-4d0b-a99b-81e0f207be63-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.679275 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.679080 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx9wt\" (UniqueName: \"kubernetes.io/projected/a03005eb-15b1-4d0b-a99b-81e0f207be63-kube-api-access-gx9wt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:46.870907 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.870872 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2"] Nov 02 14:57:46.883098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:46.883064 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-nhtj2"] Nov 02 14:57:47.206733 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:47.206696 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:47.430849 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:47.430814 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" path="/var/lib/kubelet/pods/a03005eb-15b1-4d0b-a99b-81e0f207be63/volumes" Nov 02 14:57:47.956321 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:47.956220 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:48.824625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:48.824588 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:57:49.755739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:49.755693 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:49.911693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:49.911658 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl"] Nov 02 14:57:49.939138 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:49.939104 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:49.940832 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:49.940812 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm"] Nov 02 14:57:50.001658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.001621 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" podUID="ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7" containerName="payload-pre-processing" containerID="cri-o://69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3" gracePeriod=25 Nov 02 14:57:50.118028 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.117902 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:50.296911 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.296883 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:57:50.297408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.297385 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:50.419518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.419485 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-plugins-config-volume\") pod \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\" (UID: \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\") " Nov 02 14:57:50.419719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.419632 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-kube-api-access-8fc5l\" (UniqueName: \"kubernetes.io/projected/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-kube-api-access-8fc5l\") pod \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\" (UID: \"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7\") " Nov 02 14:57:50.419895 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.419868 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-plugins-config-volume" pod "ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7" (UID: "ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:50.420046 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.420028 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:50.422036 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.422007 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-kube-api-access-8fc5l" pod "ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7" (UID: "ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7"). InnerVolumeSpecName "kube-api-access-8fc5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:50.526267 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.525242 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fc5l\" (UniqueName: \"kubernetes.io/projected/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7-kube-api-access-8fc5l\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:50.533226 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.532977 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:50.549547 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.549503 5591 generic.go:356] "Generic (PLEG): container finished" podID="ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7" containerID="69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3" exitCode=0 Nov 02 14:57:50.550760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.549570 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" Nov 02 14:57:50.550760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.549603 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" event={"ID":"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7","Type":"ContainerDied","Data":"69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3"} Nov 02 14:57:50.550760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.549658 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd" event={"ID":"ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7","Type":"ContainerDied","Data":"dd2b85a4dc2acfae5e1713472dc29612829c07be9cd76b9a4adfd68e7d50beaf"} Nov 02 14:57:50.550760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.549691 5591 scope.go:122] "RemoveContainer" containerID="69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3" Nov 02 14:57:50.597943 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.597916 5591 scope.go:122] "RemoveContainer" containerID="69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3" Nov 02 14:57:50.598368 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:50.598343 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3\": container with ID starting with 69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3 not found: ID does not exist" containerID="69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3" Nov 02 14:57:50.598486 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.598375 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3"} err="failed to get container status \"69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3\": rpc error: code = NotFound desc = could not find container \"69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3\": container with ID starting with 69f04a40434720cdf4cbf5fcb879a211c6dbd5e29473f3bf70f25110199508b3 not found: ID does not exist" Nov 02 14:57:50.611742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.611713 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd"] Nov 02 14:57:50.613782 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.613755 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0d1ca7-f757679f9-5sprd"] Nov 02 14:57:50.727743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.727712 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:57:50.772709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.772681 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:57:50.829669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.829591 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-kube-api-access-7t6sk\" (UniqueName: \"kubernetes.io/projected/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-kube-api-access-7t6sk\") pod \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\" (UID: \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\") " Nov 02 14:57:50.829669 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.829632 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-plugins-config-volume\") pod \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\" (UID: \"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01\") " Nov 02 14:57:50.830176 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.830147 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-plugins-config-volume" pod "ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" (UID: "ae397ae7-a1b3-4f5b-be33-cbc33d08bb01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:50.831862 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.831838 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-kube-api-access-7t6sk" pod "ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" (UID: "ae397ae7-a1b3-4f5b-be33-cbc33d08bb01"). InnerVolumeSpecName "kube-api-access-7t6sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:50.930779 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.930742 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4a5237-6a6a-4017-8107-f011ce4fe99e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a5237-6a6a-4017-8107-f011ce4fe99e-plugins-config-volume\") pod \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\" (UID: \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\") " Nov 02 14:57:50.931294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.930928 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4a5237-6a6a-4017-8107-f011ce4fe99e-kube-api-access-k7z9t\" (UniqueName: \"kubernetes.io/projected/ba4a5237-6a6a-4017-8107-f011ce4fe99e-kube-api-access-k7z9t\") pod \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\" (UID: \"ba4a5237-6a6a-4017-8107-f011ce4fe99e\") " Nov 02 14:57:50.931294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.931153 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4a5237-6a6a-4017-8107-f011ce4fe99e-plugins-config-volume" pod "ba4a5237-6a6a-4017-8107-f011ce4fe99e" (UID: "ba4a5237-6a6a-4017-8107-f011ce4fe99e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:50.931470 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.931448 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t6sk\" (UniqueName: \"kubernetes.io/projected/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-kube-api-access-7t6sk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:50.931538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.931475 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:50.931538 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.931491 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a5237-6a6a-4017-8107-f011ce4fe99e-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:50.932960 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:50.932938 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a5237-6a6a-4017-8107-f011ce4fe99e-kube-api-access-k7z9t" pod "ba4a5237-6a6a-4017-8107-f011ce4fe99e" (UID: "ba4a5237-6a6a-4017-8107-f011ce4fe99e"). InnerVolumeSpecName "kube-api-access-k7z9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:51.032409 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.032378 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7z9t\" (UniqueName: \"kubernetes.io/projected/ba4a5237-6a6a-4017-8107-f011ce4fe99e-kube-api-access-k7z9t\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:51.223898 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.223861 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6"] Nov 02 14:57:51.224144 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.224118 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" podUID="3e862b91-b6c8-44b6-b8d3-ca45d890af9d" containerName="maas-api" containerID="cri-o://f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a" gracePeriod=30 Nov 02 14:57:51.432702 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.432665 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7" path="/var/lib/kubelet/pods/ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7/volumes" Nov 02 14:57:51.539274 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.539246 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:57:51.558715 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.558678 5591 generic.go:356] "Generic (PLEG): container finished" podID="ba4a5237-6a6a-4017-8107-f011ce4fe99e" containerID="dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687" exitCode=0 Nov 02 14:57:51.558920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.558735 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" Nov 02 14:57:51.558920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.558764 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" event={"ID":"ba4a5237-6a6a-4017-8107-f011ce4fe99e","Type":"ContainerDied","Data":"dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687"} Nov 02 14:57:51.558920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.558824 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh" event={"ID":"ba4a5237-6a6a-4017-8107-f011ce4fe99e","Type":"ContainerDied","Data":"ae31b1b43662fa5529fe4175811e5e1284874f2677dd0019bf7b842f006f4182"} Nov 02 14:57:51.558920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.558854 5591 scope.go:122] "RemoveContainer" containerID="dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687" Nov 02 14:57:51.560243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.560218 5591 generic.go:356] "Generic (PLEG): container finished" podID="3e862b91-b6c8-44b6-b8d3-ca45d890af9d" containerID="f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a" exitCode=0 Nov 02 14:57:51.560390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.560258 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" Nov 02 14:57:51.560390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.560304 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" event={"ID":"3e862b91-b6c8-44b6-b8d3-ca45d890af9d","Type":"ContainerDied","Data":"f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a"} Nov 02 14:57:51.560390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.560353 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6" event={"ID":"3e862b91-b6c8-44b6-b8d3-ca45d890af9d","Type":"ContainerDied","Data":"7dbb8fbd3842df8f698fc0ba186f62a7fb9bc14a09cfe717adc5bbd17d99fcb5"} Nov 02 14:57:51.561857 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.561836 5591 generic.go:356] "Generic (PLEG): container finished" podID="ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" containerID="1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1" exitCode=0 Nov 02 14:57:51.561937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.561869 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" event={"ID":"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01","Type":"ContainerDied","Data":"1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1"} Nov 02 14:57:51.561937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.561878 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" Nov 02 14:57:51.561937 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.561900 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7" event={"ID":"ae397ae7-a1b3-4f5b-be33-cbc33d08bb01","Type":"ContainerDied","Data":"1b30812d1da2607a56a6dd118e97abdfc0355ae4c7b60ce9124f1daa963f214b"} Nov 02 14:57:51.569093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.569075 5591 scope.go:122] "RemoveContainer" containerID="dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687" Nov 02 14:57:51.569354 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:51.569329 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687\": container with ID starting with dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687 not found: ID does not exist" containerID="dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687" Nov 02 14:57:51.569476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.569358 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687"} err="failed to get container status \"dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687\": rpc error: code = NotFound desc = could not find container \"dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687\": container with ID starting with dcc84b972255dcae8092e9f14d1a20aec579a7af35011d025b98afa4dda96687 not found: ID does not exist" Nov 02 14:57:51.569476 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.569377 5591 scope.go:122] "RemoveContainer" containerID="f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a" Nov 02 14:57:51.577751 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.577730 5591 scope.go:122] "RemoveContainer" containerID="f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a" Nov 02 14:57:51.578071 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:51.578050 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a\": container with ID starting with f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a not found: ID does not exist" containerID="f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a" Nov 02 14:57:51.578169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.578077 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a"} err="failed to get container status \"f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a\": rpc error: code = NotFound desc = could not find container \"f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a\": container with ID starting with f0cbff945a1bb92a824214d05b56335d4f74bb9b5bb80c19e85aba3b75748a1a not found: ID does not exist" Nov 02 14:57:51.578169 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.578095 5591 scope.go:122] "RemoveContainer" containerID="1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1" Nov 02 14:57:51.587387 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.587355 5591 scope.go:122] "RemoveContainer" containerID="1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1" Nov 02 14:57:51.587735 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:51.587705 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1\": container with ID starting with 1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1 not found: ID does not exist" containerID="1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1" Nov 02 14:57:51.588022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.587744 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1"} err="failed to get container status \"1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1\": rpc error: code = NotFound desc = could not find container \"1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1\": container with ID starting with 1524f6d0d7bd0124835934ab4e424e1d07d429b22b798d19dc5666adb8ea43f1 not found: ID does not exist" Nov 02 14:57:51.635535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.635502 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7"] Nov 02 14:57:51.639613 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.639585 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-metrics-certs\") pod \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " Nov 02 14:57:51.639764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.639628 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-kube-api-access-ppv84\" (UniqueName: \"kubernetes.io/projected/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-kube-api-access-ppv84\") pod \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " Nov 02 14:57:51.639764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.639679 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-maas-api-tls\") pod \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " Nov 02 14:57:51.639764 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.639714 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-openshift-service-ca\") pod \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\" (UID: \"3e862b91-b6c8-44b6-b8d3-ca45d890af9d\") " Nov 02 14:57:51.640193 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.640168 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-openshift-service-ca" pod "3e862b91-b6c8-44b6-b8d3-ca45d890af9d" (UID: "3e862b91-b6c8-44b6-b8d3-ca45d890af9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:51.640357 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.640330 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2cddf2-6f568dd759-njms7"] Nov 02 14:57:51.641860 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.641841 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-maas-api-tls" pod "3e862b91-b6c8-44b6-b8d3-ca45d890af9d" (UID: "3e862b91-b6c8-44b6-b8d3-ca45d890af9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:51.642256 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.642233 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-metrics-certs" pod "3e862b91-b6c8-44b6-b8d3-ca45d890af9d" (UID: "3e862b91-b6c8-44b6-b8d3-ca45d890af9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:51.642327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.642266 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-kube-api-access-ppv84" pod "3e862b91-b6c8-44b6-b8d3-ca45d890af9d" (UID: "3e862b91-b6c8-44b6-b8d3-ca45d890af9d"). InnerVolumeSpecName "kube-api-access-ppv84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:51.675523 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.675485 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh"] Nov 02 14:57:51.681153 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.681121 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2cddf2-dc85fbd47-9w6dh"] Nov 02 14:57:51.750016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.749941 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:51.750016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.750003 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:51.750016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.750020 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:51.750350 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.750036 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppv84\" (UniqueName: \"kubernetes.io/projected/3e862b91-b6c8-44b6-b8d3-ca45d890af9d-kube-api-access-ppv84\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:51.912469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.912383 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6"] Nov 02 14:57:51.916035 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:51.915998 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d1ca7-84dbdd5d8d-qcps6"] Nov 02 14:57:52.174570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:52.174522 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:52.240446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:52.240408 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m"] Nov 02 14:57:52.240672 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:52.240643 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" containerID="cri-o://5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4" gracePeriod=30 Nov 02 14:57:52.347286 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:52.344507 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:53.429171 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:53.429088 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e862b91-b6c8-44b6-b8d3-ca45d890af9d" path="/var/lib/kubelet/pods/3e862b91-b6c8-44b6-b8d3-ca45d890af9d/volumes" Nov 02 14:57:53.429526 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:53.429475 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" path="/var/lib/kubelet/pods/ae397ae7-a1b3-4f5b-be33-cbc33d08bb01/volumes" Nov 02 14:57:53.429851 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:53.429837 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4a5237-6a6a-4017-8107-f011ce4fe99e" path="/var/lib/kubelet/pods/ba4a5237-6a6a-4017-8107-f011ce4fe99e/volumes" Nov 02 14:57:53.680956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:53.680871 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:54.366567 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:54.366522 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:54.526522 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:54.526488 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:54.912146 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:54.912097 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" podUID="c10e334b-2678-4fe5-a5da-16d9bd9890eb" containerName="payload-processing" containerID="cri-o://3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66" gracePeriod=25 Nov 02 14:57:54.941307 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:54.941262 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" podUID="e254d3cf-7490-4cf8-b709-493a2ee68794" containerName="payload-pre-processing" containerID="cri-o://fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932" gracePeriod=25 Nov 02 14:57:55.283112 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:55.283026 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:55.591922 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:55.591826 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2"] Nov 02 14:57:55.592576 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:55.592084 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" containerID="cri-o://82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c" gracePeriod=30 Nov 02 14:57:55.711088 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:55.711032 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:56.159356 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:56.159320 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:56.329054 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:56.329017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:56.640649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:56.640606 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:56.819187 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:56.819135 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:57.441413 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.441386 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:57.521424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.521366 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:57:57.526438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526417 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-kube-api-access-5j5hd\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-kube-api-access-5j5hd\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526456 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-data\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526481 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-certs\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526565 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526533 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/253450c0-3bb5-4b4a-a312-63491e337372-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253450c0-3bb5-4b4a-a312-63491e337372-istio-podinfo\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526566 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-credential-socket\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526601 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-istio-token\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-istio-token\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526673 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253450c0-3bb5-4b4a-a312-63491e337372-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253450c0-3bb5-4b4a-a312-63491e337372-istiod-ca-cert\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526699 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-socket\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526761 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-envoy\") pod \"253450c0-3bb5-4b4a-a312-63491e337372\" (UID: \"253450c0-3bb5-4b4a-a312-63491e337372\") " Nov 02 14:57:57.526974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526787 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-data" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:57.526974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526837 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-certs" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:57.526974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.526929 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-credential-socket" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:57.527216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.527059 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253450c0-3bb5-4b4a-a312-63491e337372-istiod-ca-cert" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:57.527280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.527237 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-socket" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:57.527280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.527247 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.527280 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.527268 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.527473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.527282 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.527473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.527296 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253450c0-3bb5-4b4a-a312-63491e337372-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.528756 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.528721 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/253450c0-3bb5-4b4a-a312-63491e337372-istio-podinfo" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:57:57.528854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.528773 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-kube-api-access-5j5hd" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "kube-api-access-5j5hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:57.528854 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.528786 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-envoy" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:57:57.529101 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.529084 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-istio-token" pod "253450c0-3bb5-4b4a-a312-63491e337372" (UID: "253450c0-3bb5-4b4a-a312-63491e337372"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:57.629478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.629437 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j5hd\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-kube-api-access-5j5hd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.629478 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.629472 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253450c0-3bb5-4b4a-a312-63491e337372-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.629717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.629486 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253450c0-3bb5-4b4a-a312-63491e337372-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.629717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.629499 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.629717 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.629511 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253450c0-3bb5-4b4a-a312-63491e337372-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:57:57.636552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.636231 5591 generic.go:356] "Generic (PLEG): container finished" podID="253450c0-3bb5-4b4a-a312-63491e337372" containerID="5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4" exitCode=0 Nov 02 14:57:57.636552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.636314 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" Nov 02 14:57:57.636552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.636331 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" event={"ID":"253450c0-3bb5-4b4a-a312-63491e337372","Type":"ContainerDied","Data":"5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4"} Nov 02 14:57:57.636552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.636387 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m" event={"ID":"253450c0-3bb5-4b4a-a312-63491e337372","Type":"ContainerDied","Data":"93df1760a46e0f370a8ff059fc5280286cbe3df4bd4f12c44fe699b08bac9d77"} Nov 02 14:57:57.636552 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.636420 5591 scope.go:122] "RemoveContainer" containerID="5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4" Nov 02 14:57:57.675271 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.675234 5591 scope.go:122] "RemoveContainer" containerID="d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b" Nov 02 14:57:57.677761 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.677733 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw"] Nov 02 14:57:57.677956 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.677931 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" containerID="cri-o://5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565" gracePeriod=30 Nov 02 14:57:57.686308 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.686277 5591 scope.go:122] "RemoveContainer" containerID="5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4" Nov 02 14:57:57.686639 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:57.686616 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4\": container with ID starting with 5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4 not found: ID does not exist" containerID="5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4" Nov 02 14:57:57.686716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.686648 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4"} err="failed to get container status \"5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4\": rpc error: code = NotFound desc = could not find container \"5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4\": container with ID starting with 5089b74fd514f5530e09971477ced04159448ebf1c216ffb707a577f50c598d4 not found: ID does not exist" Nov 02 14:57:57.686716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.686666 5591 scope.go:122] "RemoveContainer" containerID="d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b" Nov 02 14:57:57.687007 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:57:57.686960 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b\": container with ID starting with d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b not found: ID does not exist" containerID="d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b" Nov 02 14:57:57.687083 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.687020 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b"} err="failed to get container status \"d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b\": rpc error: code = NotFound desc = could not find container \"d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b\": container with ID starting with d458cae5445eb9e2fba6f522610415fcc802b2ed707cbf17c56af3674b040a6b not found: ID does not exist" Nov 02 14:57:57.699196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.699149 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m"] Nov 02 14:57:57.702969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.702944 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-7cm9m"] Nov 02 14:57:57.789969 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:57.787668 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:58.309570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.309516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:58.552208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.552170 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:58.693294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693253 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh"] Nov 02 14:57:58.693689 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693680 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7" containerName="payload-pre-processing" Nov 02 14:57:58.693737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693691 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7" containerName="payload-pre-processing" Nov 02 14:57:58.693737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693708 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ba4a5237-6a6a-4017-8107-f011ce4fe99e" containerName="payload-pre-processing" Nov 02 14:57:58.693737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693713 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4a5237-6a6a-4017-8107-f011ce4fe99e" containerName="payload-pre-processing" Nov 02 14:57:58.693737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693726 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" Nov 02 14:57:58.693737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693732 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693742 5591 cpu_manager.go:414] "containerMap: removing container" podUID="3e862b91-b6c8-44b6-b8d3-ca45d890af9d" containerName="maas-api" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693747 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e862b91-b6c8-44b6-b8d3-ca45d890af9d" containerName="maas-api" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693757 5591 cpu_manager.go:414] "containerMap: removing container" podUID="afa63769-17d5-4169-96f0-f586c739a12e" containerName="maas-api" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693762 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa63769-17d5-4169-96f0-f586c739a12e" containerName="maas-api" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693772 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693776 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693784 5591 cpu_manager.go:414] "containerMap: removing container" podUID="43b2c8e3-58b7-447b-b179-1e83d6543efe" containerName="maas-api" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693789 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b2c8e3-58b7-447b-b179-1e83d6543efe" containerName="maas-api" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693798 5591 cpu_manager.go:414] "containerMap: removing container" podUID="39540df0-9900-4ded-8dd2-d21ed94e1dbc" containerName="main" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693803 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39540df0-9900-4ded-8dd2-d21ed94e1dbc" containerName="main" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693810 5591 cpu_manager.go:414] "containerMap: removing container" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693815 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693824 5591 cpu_manager.go:414] "containerMap: removing container" podUID="13100993-e396-430e-9692-694b284426fc" containerName="payload-processing" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693829 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13100993-e396-430e-9692-694b284426fc" containerName="payload-processing" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693836 5591 cpu_manager.go:414] "containerMap: removing container" podUID="40992517-7563-4c5a-8395-8f560b3042ec" containerName="test-curl-7033-ee22c5" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693840 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40992517-7563-4c5a-8395-8f560b3042ec" containerName="test-curl-7033-ee22c5" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693848 5591 cpu_manager.go:414] "containerMap: removing container" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693852 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693860 5591 cpu_manager.go:414] "containerMap: removing container" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693864 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693871 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" containerName="payload-processing" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693876 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" containerName="payload-processing" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693883 5591 cpu_manager.go:414] "containerMap: removing container" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693888 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693899 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bdde9b7a-bec6-4c34-a647-2448a0301ff6" containerName="payload-processing" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693904 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdde9b7a-bec6-4c34-a647-2448a0301ff6" containerName="payload-processing" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693913 5591 cpu_manager.go:414] "containerMap: removing container" podUID="40d0620c-2792-4b2e-8c86-367582a82e2c" containerName="payload-processing" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693917 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d0620c-2792-4b2e-8c86-367582a82e2c" containerName="payload-processing" Nov 02 14:57:58.693913 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693923 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693928 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693935 5591 cpu_manager.go:414] "containerMap: removing container" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.693939 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694018 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694026 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03005eb-15b1-4d0b-a99b-81e0f207be63" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694034 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae397ae7-a1b3-4f5b-be33-cbc33d08bb01" containerName="payload-processing" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694040 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694047 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdde9b7a-bec6-4c34-a647-2448a0301ff6" containerName="payload-processing" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694052 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d0620c-2792-4b2e-8c86-367582a82e2c" containerName="payload-processing" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694058 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694064 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="98702400-9887-4120-adea-809091fdf0aa" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694069 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3d1492-01c4-453d-bcb9-4a18c0fbe6d7" containerName="payload-pre-processing" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694074 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4a5237-6a6a-4017-8107-f011ce4fe99e" containerName="payload-pre-processing" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694081 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694086 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e862b91-b6c8-44b6-b8d3-ca45d890af9d" containerName="maas-api" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694092 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa63769-17d5-4169-96f0-f586c739a12e" containerName="maas-api" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694099 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf95dcd7-f3e2-4ced-ac82-9e0e58513b3c" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694105 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b2c8e3-58b7-447b-b179-1e83d6543efe" containerName="maas-api" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694111 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="39540df0-9900-4ded-8dd2-d21ed94e1dbc" containerName="main" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694117 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="253450c0-3bb5-4b4a-a312-63491e337372" containerName="istio-proxy" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694124 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="13100993-e396-430e-9692-694b284426fc" containerName="payload-processing" Nov 02 14:57:58.694848 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.694129 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="40992517-7563-4c5a-8395-8f560b3042ec" containerName="test-curl-7033-ee22c5" Nov 02 14:57:58.698504 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.698486 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.715763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.715736 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh"] Nov 02 14:57:58.732396 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.732365 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:57:58.732533 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.732446 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\"" Nov 02 14:57:58.738532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.738505 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.738659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.738558 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.738659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.738644 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gjd\" (UniqueName: \"kubernetes.io/projected/78edfcb3-0a81-4845-925c-dc30b7fac4ba-kube-api-access-98gjd\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.839238 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.839205 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.839422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.839349 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.839422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.839372 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gjd\" (UniqueName: \"kubernetes.io/projected/78edfcb3-0a81-4845-925c-dc30b7fac4ba-kube-api-access-98gjd\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.839856 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.839834 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.841731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.841710 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:58.862287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:58.862259 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gjd\" (UniqueName: \"kubernetes.io/projected/78edfcb3-0a81-4845-925c-dc30b7fac4ba-kube-api-access-98gjd\") pod \"maas-api-revoke-sub-47b5d118-grcqh\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:59.012141 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:59.012065 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:57:59.313543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:59.313511 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh"] Nov 02 14:57:59.315427 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:57:59.315398 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78edfcb3_0a81_4845_925c_dc30b7fac4ba.slice/crio-ef5f69436fdca34972c9f4a0bfe53e579d28bdb4475d4adb4e9674c38d14f9d8 WatchSource:0}: Error finding container ef5f69436fdca34972c9f4a0bfe53e579d28bdb4475d4adb4e9674c38d14f9d8: Status 404 returned error can't find the container with id ef5f69436fdca34972c9f4a0bfe53e579d28bdb4475d4adb4e9674c38d14f9d8 Nov 02 14:57:59.430622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:59.430588 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253450c0-3bb5-4b4a-a312-63491e337372" path="/var/lib/kubelet/pods/253450c0-3bb5-4b4a-a312-63491e337372/volumes" Nov 02 14:57:59.649500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:57:59.649469 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" event={"ID":"78edfcb3-0a81-4845-925c-dc30b7fac4ba","Type":"ContainerStarted","Data":"ef5f69436fdca34972c9f4a0bfe53e579d28bdb4475d4adb4e9674c38d14f9d8"} Nov 02 14:58:00.167324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.167220 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:00.655967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.655932 5591 generic.go:356] "Generic (PLEG): container finished" podID="78edfcb3-0a81-4845-925c-dc30b7fac4ba" containerID="f08c1611e0ee3d24e241be8ba550e94e3c4e97925d547c59414a329e38c0e4c8" exitCode=0 Nov 02 14:58:00.656140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.656019 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" event={"ID":"78edfcb3-0a81-4845-925c-dc30b7fac4ba","Type":"ContainerDied","Data":"f08c1611e0ee3d24e241be8ba550e94e3c4e97925d547c59414a329e38c0e4c8"} Nov 02 14:58:00.838415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.838388 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:58:00.857428 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857351 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-token\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-token\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.857581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857430 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-envoy\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.857581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857526 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-podinfo\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.857718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857588 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ee53c2-ff7a-496e-be14-6e2e938be315-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ee53c2-ff7a-496e-be14-6e2e938be315-istiod-ca-cert\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.857718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857691 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-credential-socket\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.857831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857747 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-data\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.857831 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857780 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-kube-api-access-b6nkn\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-kube-api-access-b6nkn\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.857936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857841 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-certs\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.857936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.857870 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-socket\") pod \"62ee53c2-ff7a-496e-be14-6e2e938be315\" (UID: \"62ee53c2-ff7a-496e-be14-6e2e938be315\") " Nov 02 14:58:00.858086 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858038 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-credential-socket" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:00.858145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858093 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-data" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:00.858145 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858125 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ee53c2-ff7a-496e-be14-6e2e938be315-istiod-ca-cert" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:00.858289 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858246 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-socket" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:00.858355 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858315 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-certs" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:00.858414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858358 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ee53c2-ff7a-496e-be14-6e2e938be315-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:00.858414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858379 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:00.858414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858395 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:00.858414 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.858409 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:00.860011 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.859964 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-podinfo" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:58:00.860128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.860059 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-envoy" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:00.860128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.860070 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-token" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:00.860254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.860146 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-kube-api-access-b6nkn" pod "62ee53c2-ff7a-496e-be14-6e2e938be315" (UID: "62ee53c2-ff7a-496e-be14-6e2e938be315"). InnerVolumeSpecName "kube-api-access-b6nkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:00.959490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.959450 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:00.959490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.959485 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:00.959490 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.959496 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ee53c2-ff7a-496e-be14-6e2e938be315-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:00.959737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.959505 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6nkn\" (UniqueName: \"kubernetes.io/projected/62ee53c2-ff7a-496e-be14-6e2e938be315-kube-api-access-b6nkn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:00.959737 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:00.959515 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ee53c2-ff7a-496e-be14-6e2e938be315-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:01.068209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.068176 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz"] Nov 02 14:58:01.073721 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:01.073684 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:58:01.087463 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.087428 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr"] Nov 02 14:58:01.267754 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.267702 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:01.662864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.662827 5591 generic.go:356] "Generic (PLEG): container finished" podID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerID="82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c" exitCode=0 Nov 02 14:58:01.663085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.662884 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" Nov 02 14:58:01.663085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.662916 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" event={"ID":"62ee53c2-ff7a-496e-be14-6e2e938be315","Type":"ContainerDied","Data":"82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c"} Nov 02 14:58:01.663085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.663006 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2" event={"ID":"62ee53c2-ff7a-496e-be14-6e2e938be315","Type":"ContainerDied","Data":"54c330021615271ab389d8d4a0c14c7c30413e449e69158c5ecb1d148f8c94b0"} Nov 02 14:58:01.663085 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.663019 5591 scope.go:122] "RemoveContainer" containerID="82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c" Nov 02 14:58:01.672976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.672954 5591 scope.go:122] "RemoveContainer" containerID="82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c" Nov 02 14:58:01.673352 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:01.673330 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c\": container with ID starting with 82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c not found: ID does not exist" containerID="82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c" Nov 02 14:58:01.673417 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.673364 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c"} err="failed to get container status \"82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c\": rpc error: code = NotFound desc = could not find container \"82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c\": container with ID starting with 82d8e06c62e6304a392544d8e74e16b68c7f19102f5a5c7244f52d97c5bfed9c not found: ID does not exist" Nov 02 14:58:01.714863 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.714825 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2"] Nov 02 14:58:01.727469 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.727439 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-hppz2"] Nov 02 14:58:01.819704 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.819666 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:58:01.868742 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.868709 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-xks-ca\") pod \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " Nov 02 14:58:01.868939 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.868924 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78edfcb3-0a81-4845-925c-dc30b7fac4ba-kube-api-access-98gjd\" (UniqueName: \"kubernetes.io/projected/78edfcb3-0a81-4845-925c-dc30b7fac4ba-kube-api-access-98gjd\") pod \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " Nov 02 14:58:01.868995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.868972 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-openshift-ca\") pod \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\" (UID: \"78edfcb3-0a81-4845-925c-dc30b7fac4ba\") " Nov 02 14:58:01.869390 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.869360 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-openshift-ca" pod "78edfcb3-0a81-4845-925c-dc30b7fac4ba" (UID: "78edfcb3-0a81-4845-925c-dc30b7fac4ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:01.870959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.870930 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-xks-ca" pod "78edfcb3-0a81-4845-925c-dc30b7fac4ba" (UID: "78edfcb3-0a81-4845-925c-dc30b7fac4ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:01.871513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.871494 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78edfcb3-0a81-4845-925c-dc30b7fac4ba-kube-api-access-98gjd" pod "78edfcb3-0a81-4845-925c-dc30b7fac4ba" (UID: "78edfcb3-0a81-4845-925c-dc30b7fac4ba"). InnerVolumeSpecName "kube-api-access-98gjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:01.933612 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.924520 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:01.970262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.970223 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:01.970262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.970256 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78edfcb3-0a81-4845-925c-dc30b7fac4ba-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:01.970262 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:01.970267 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98gjd\" (UniqueName: \"kubernetes.io/projected/78edfcb3-0a81-4845-925c-dc30b7fac4ba-kube-api-access-98gjd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:02.276517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.276435 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:02.671304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.671271 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" Nov 02 14:58:02.671304 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.671287 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh" event={"ID":"78edfcb3-0a81-4845-925c-dc30b7fac4ba","Type":"ContainerDied","Data":"ef5f69436fdca34972c9f4a0bfe53e579d28bdb4475d4adb4e9674c38d14f9d8"} Nov 02 14:58:02.671549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.671350 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef5f69436fdca34972c9f4a0bfe53e579d28bdb4475d4adb4e9674c38d14f9d8" Nov 02 14:58:02.817209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.817164 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" podUID="e254d3cf-7490-4cf8-b709-493a2ee68794" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:58:02.939213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.939187 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:58:02.981162 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981129 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-socket\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981181 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-certs\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981213 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-data\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981323 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981272 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-credential-socket\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981370 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-istio-token\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-istio-token\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981412 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-socket" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:02.981503 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981415 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-envoy\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981503 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-data" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:02.981664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981509 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2574d71-5766-4983-9fbd-41c3103170e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2574d71-5766-4983-9fbd-41c3103170e4-istio-podinfo\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981531 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-certs" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:02.981664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981569 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-kube-api-access-qdcpf\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-kube-api-access-qdcpf\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981621 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2574d71-5766-4983-9fbd-41c3103170e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2574d71-5766-4983-9fbd-41c3103170e4-istiod-ca-cert\") pod \"d2574d71-5766-4983-9fbd-41c3103170e4\" (UID: \"d2574d71-5766-4983-9fbd-41c3103170e4\") " Nov 02 14:58:02.981906 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.981702 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-credential-socket" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:02.982065 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.982042 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:02.982197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.982157 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:02.982197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.982175 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:02.982197 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.982187 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:02.982339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.982101 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2574d71-5766-4983-9fbd-41c3103170e4-istiod-ca-cert" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:02.983693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.983646 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-envoy" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:02.983693 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.983678 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2574d71-5766-4983-9fbd-41c3103170e4-istio-podinfo" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:58:02.984016 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.983975 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-kube-api-access-qdcpf" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "kube-api-access-qdcpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:02.984114 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:02.984020 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-istio-token" pod "d2574d71-5766-4983-9fbd-41c3103170e4" (UID: "d2574d71-5766-4983-9fbd-41c3103170e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:03.083202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.083167 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:03.083202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.083199 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2574d71-5766-4983-9fbd-41c3103170e4-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:03.083202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.083209 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2574d71-5766-4983-9fbd-41c3103170e4-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:03.083438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.083217 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdcpf\" (UniqueName: \"kubernetes.io/projected/d2574d71-5766-4983-9fbd-41c3103170e4-kube-api-access-qdcpf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:03.083438 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.083228 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2574d71-5766-4983-9fbd-41c3103170e4-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:03.153049 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.153015 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:03.385484 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.385430 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:03.430202 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.430169 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" path="/var/lib/kubelet/pods/62ee53c2-ff7a-496e-be14-6e2e938be315/volumes" Nov 02 14:58:03.679833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.679714 5591 generic.go:356] "Generic (PLEG): container finished" podID="d2574d71-5766-4983-9fbd-41c3103170e4" containerID="5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565" exitCode=0 Nov 02 14:58:03.679833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.679769 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" Nov 02 14:58:03.679833 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.679791 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" event={"ID":"d2574d71-5766-4983-9fbd-41c3103170e4","Type":"ContainerDied","Data":"5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565"} Nov 02 14:58:03.680163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.679856 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw" event={"ID":"d2574d71-5766-4983-9fbd-41c3103170e4","Type":"ContainerDied","Data":"343ba3091103a89eeb940c909cdb9dc91134d189dabd8f8c9bda06a351704500"} Nov 02 14:58:03.680163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.679879 5591 scope.go:122] "RemoveContainer" containerID="5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565" Nov 02 14:58:03.687196 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.687151 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:03.689100 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.689081 5591 scope.go:122] "RemoveContainer" containerID="5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565" Nov 02 14:58:03.689357 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:03.689338 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565\": container with ID starting with 5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565 not found: ID does not exist" containerID="5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565" Nov 02 14:58:03.689408 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.689365 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565"} err="failed to get container status \"5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565\": rpc error: code = NotFound desc = could not find container \"5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565\": container with ID starting with 5f9669535dcb9366ac84b53e955a7350b8916ffcee086d0c701947377ee98565 not found: ID does not exist" Nov 02 14:58:03.751923 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.751890 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw"] Nov 02 14:58:03.763571 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.763543 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-x7bhw"] Nov 02 14:58:03.825389 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.825353 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" podUID="c10e334b-2678-4fe5-a5da-16d9bd9890eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:58:03.907420 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:03.907377 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:04.231841 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:04.231804 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:05.430338 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:05.430301 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" path="/var/lib/kubelet/pods/d2574d71-5766-4983-9fbd-41c3103170e4/volumes" Nov 02 14:58:05.558644 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:05.558611 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:07.334607 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:07.334552 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:07.673453 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:07.673422 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:07.853302 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:07.853262 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:08.445692 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:08.445655 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:58:09.197359 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:09.197330 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:09.530018 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:09.529914 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:10.135561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.135531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:10.300649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.300616 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:10.466540 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.466461 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:10.815241 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815150 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5"] Nov 02 14:58:10.815611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815569 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" Nov 02 14:58:10.815611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815581 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" Nov 02 14:58:10.815611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815606 5591 cpu_manager.go:414] "containerMap: removing container" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" Nov 02 14:58:10.815611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815611 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" Nov 02 14:58:10.815760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815622 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" Nov 02 14:58:10.815760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815628 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" Nov 02 14:58:10.815760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815640 5591 cpu_manager.go:414] "containerMap: removing container" podUID="78edfcb3-0a81-4845-925c-dc30b7fac4ba" containerName="revoke-keys" Nov 02 14:58:10.815760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815645 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78edfcb3-0a81-4845-925c-dc30b7fac4ba" containerName="revoke-keys" Nov 02 14:58:10.815760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815702 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ee53c2-ff7a-496e-be14-6e2e938be315" containerName="istio-proxy" Nov 02 14:58:10.815760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815713 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2574d71-5766-4983-9fbd-41c3103170e4" containerName="istio-proxy" Nov 02 14:58:10.815760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815724 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="78edfcb3-0a81-4845-925c-dc30b7fac4ba" containerName="revoke-keys" Nov 02 14:58:10.815760 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.815735 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39babc9-1aa2-4878-8b21-b98ddd3a7f34" containerName="istio-proxy" Nov 02 14:58:10.820410 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.820391 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.848579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.848551 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:58:10.848724 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.848589 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\"" Nov 02 14:58:10.850397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.850374 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5"] Nov 02 14:58:10.864378 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.864351 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxc7\" (UniqueName: \"kubernetes.io/projected/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-kube-api-access-dtxc7\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.864516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.864494 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.864617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.864601 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.965953 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.965913 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.966122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.966080 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.966122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.966109 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxc7\" (UniqueName: \"kubernetes.io/projected/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-kube-api-access-dtxc7\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.966610 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.966573 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.968426 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.968404 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:10.989775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:10.989747 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxc7\" (UniqueName: \"kubernetes.io/projected/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-kube-api-access-dtxc7\") pod \"maas-api-revoke-sub-71e1b6ea-45tg5\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:11.097776 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:11.097676 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:11.132900 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:11.132868 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:11.417563 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:11.417528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:11.431912 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:11.431888 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5"] Nov 02 14:58:11.434308 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:58:11.434275 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fc8cc1_c69e_4cb8_9bda_5370b6d65b68.slice/crio-30cbd65be736eb1a4837db70046338eae4e831af16c2c6ccc61095f35fdd0da6 WatchSource:0}: Error finding container 30cbd65be736eb1a4837db70046338eae4e831af16c2c6ccc61095f35fdd0da6: Status 404 returned error can't find the container with id 30cbd65be736eb1a4837db70046338eae4e831af16c2c6ccc61095f35fdd0da6 Nov 02 14:58:11.595220 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:11.595188 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:11.737045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:11.737013 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" event={"ID":"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68","Type":"ContainerStarted","Data":"30cbd65be736eb1a4837db70046338eae4e831af16c2c6ccc61095f35fdd0da6"} Nov 02 14:58:11.812917 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:11.812825 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:12.743668 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:12.743632 5591 generic.go:356] "Generic (PLEG): container finished" podID="e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68" containerID="2b80a85466e2b595f83daf017cb826cbe49e6d2a7fa9a80355dae37267882baf" exitCode=0 Nov 02 14:58:12.744120 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:12.743703 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" event={"ID":"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68","Type":"ContainerDied","Data":"2b80a85466e2b595f83daf017cb826cbe49e6d2a7fa9a80355dae37267882baf"} Nov 02 14:58:12.816106 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:12.816067 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" podUID="e254d3cf-7490-4cf8-b709-493a2ee68794" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:58:12.982829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:12.982791 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:13.825500 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:13.825461 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" podUID="c10e334b-2678-4fe5-a5da-16d9bd9890eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:58:13.891349 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:13.891321 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:13.996847 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:13.996801 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-kube-api-access-dtxc7\" (UniqueName: \"kubernetes.io/projected/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-kube-api-access-dtxc7\") pod \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " Nov 02 14:58:13.997075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:13.996894 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-openshift-ca\") pod \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " Nov 02 14:58:13.997075 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:13.996945 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-xks-ca\") pod \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\" (UID: \"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68\") " Nov 02 14:58:13.997358 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:13.997329 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-openshift-ca" pod "e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68" (UID: "e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:13.999066 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:13.999034 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-xks-ca" pod "e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68" (UID: "e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:13.999184 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:13.999080 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-kube-api-access-dtxc7" pod "e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68" (UID: "e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68"). InnerVolumeSpecName "kube-api-access-dtxc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:14.098648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:14.098609 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:14.098648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:14.098642 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:14.098648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:14.098651 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtxc7\" (UniqueName: \"kubernetes.io/projected/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68-kube-api-access-dtxc7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:14.517793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:14.517757 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:14.755630 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:14.755600 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" Nov 02 14:58:14.755814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:14.755633 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5" event={"ID":"e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68","Type":"ContainerDied","Data":"30cbd65be736eb1a4837db70046338eae4e831af16c2c6ccc61095f35fdd0da6"} Nov 02 14:58:14.755814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:14.755682 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cbd65be736eb1a4837db70046338eae4e831af16c2c6ccc61095f35fdd0da6" Nov 02 14:58:15.186649 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.186618 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:58:15.215835 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.215808 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:58:15.273082 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.273017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:15.314399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.314310 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10e334b-2678-4fe5-a5da-16d9bd9890eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e334b-2678-4fe5-a5da-16d9bd9890eb-plugins-config-volume\") pod \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\" (UID: \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\") " Nov 02 14:58:15.314541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.314418 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e334b-2678-4fe5-a5da-16d9bd9890eb-kube-api-access-wxccb\" (UniqueName: \"kubernetes.io/projected/c10e334b-2678-4fe5-a5da-16d9bd9890eb-kube-api-access-wxccb\") pod \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\" (UID: \"c10e334b-2678-4fe5-a5da-16d9bd9890eb\") " Nov 02 14:58:15.314541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.314517 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e254d3cf-7490-4cf8-b709-493a2ee68794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e254d3cf-7490-4cf8-b709-493a2ee68794-plugins-config-volume\") pod \"e254d3cf-7490-4cf8-b709-493a2ee68794\" (UID: \"e254d3cf-7490-4cf8-b709-493a2ee68794\") " Nov 02 14:58:15.314624 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.314571 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e254d3cf-7490-4cf8-b709-493a2ee68794-kube-api-access-nchd2\" (UniqueName: \"kubernetes.io/projected/e254d3cf-7490-4cf8-b709-493a2ee68794-kube-api-access-nchd2\") pod \"e254d3cf-7490-4cf8-b709-493a2ee68794\" (UID: \"e254d3cf-7490-4cf8-b709-493a2ee68794\") " Nov 02 14:58:15.314882 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.314718 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10e334b-2678-4fe5-a5da-16d9bd9890eb-plugins-config-volume" pod "c10e334b-2678-4fe5-a5da-16d9bd9890eb" (UID: "c10e334b-2678-4fe5-a5da-16d9bd9890eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:15.315003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.314874 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e254d3cf-7490-4cf8-b709-493a2ee68794-plugins-config-volume" pod "e254d3cf-7490-4cf8-b709-493a2ee68794" (UID: "e254d3cf-7490-4cf8-b709-493a2ee68794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:15.315003 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.314964 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e254d3cf-7490-4cf8-b709-493a2ee68794-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:15.315122 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.314977 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e334b-2678-4fe5-a5da-16d9bd9890eb-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:15.316684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.316659 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e254d3cf-7490-4cf8-b709-493a2ee68794-kube-api-access-nchd2" pod "e254d3cf-7490-4cf8-b709-493a2ee68794" (UID: "e254d3cf-7490-4cf8-b709-493a2ee68794"). InnerVolumeSpecName "kube-api-access-nchd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:15.316684 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.316681 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e334b-2678-4fe5-a5da-16d9bd9890eb-kube-api-access-wxccb" pod "c10e334b-2678-4fe5-a5da-16d9bd9890eb" (UID: "c10e334b-2678-4fe5-a5da-16d9bd9890eb"). InnerVolumeSpecName "kube-api-access-wxccb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:15.415532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.415497 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nchd2\" (UniqueName: \"kubernetes.io/projected/e254d3cf-7490-4cf8-b709-493a2ee68794-kube-api-access-nchd2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:15.415532 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.415526 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxccb\" (UniqueName: \"kubernetes.io/projected/c10e334b-2678-4fe5-a5da-16d9bd9890eb-kube-api-access-wxccb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:15.683952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.683917 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:15.762097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.762064 5591 generic.go:356] "Generic (PLEG): container finished" podID="c10e334b-2678-4fe5-a5da-16d9bd9890eb" containerID="3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66" exitCode=0 Nov 02 14:58:15.762265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.762112 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" Nov 02 14:58:15.762265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.762152 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" event={"ID":"c10e334b-2678-4fe5-a5da-16d9bd9890eb","Type":"ContainerDied","Data":"3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66"} Nov 02 14:58:15.762265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.762216 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl" event={"ID":"c10e334b-2678-4fe5-a5da-16d9bd9890eb","Type":"ContainerDied","Data":"1e3469dc0466101a4f506e19ccb34fc787e776710c03ecc315c2bdaac87a6e0c"} Nov 02 14:58:15.762265 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.762251 5591 scope.go:122] "RemoveContainer" containerID="3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66" Nov 02 14:58:15.763616 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.763591 5591 generic.go:356] "Generic (PLEG): container finished" podID="e254d3cf-7490-4cf8-b709-493a2ee68794" containerID="fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932" exitCode=0 Nov 02 14:58:15.763780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.763650 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" Nov 02 14:58:15.763780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.763709 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" event={"ID":"e254d3cf-7490-4cf8-b709-493a2ee68794","Type":"ContainerDied","Data":"fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932"} Nov 02 14:58:15.763780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.763761 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm" event={"ID":"e254d3cf-7490-4cf8-b709-493a2ee68794","Type":"ContainerDied","Data":"0c5e33090fc8bdc89c4ebf3832225e512a4a8747225144411e8e8afebe7103d3"} Nov 02 14:58:15.771215 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.771197 5591 scope.go:122] "RemoveContainer" containerID="3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66" Nov 02 14:58:15.771499 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:15.771477 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66\": container with ID starting with 3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66 not found: ID does not exist" containerID="3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66" Nov 02 14:58:15.771546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.771507 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66"} err="failed to get container status \"3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66\": rpc error: code = NotFound desc = could not find container \"3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66\": container with ID starting with 3670331934a7bccbe25e765cca82a779fbe37f1dda53ddd319b9e7e889130f66 not found: ID does not exist" Nov 02 14:58:15.771546 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.771524 5591 scope.go:122] "RemoveContainer" containerID="fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932" Nov 02 14:58:15.780001 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.779820 5591 scope.go:122] "RemoveContainer" containerID="fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932" Nov 02 14:58:15.780149 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:15.780128 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932\": container with ID starting with fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932 not found: ID does not exist" containerID="fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932" Nov 02 14:58:15.780209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.780161 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932"} err="failed to get container status \"fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932\": rpc error: code = NotFound desc = could not find container \"fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932\": container with ID starting with fc2498ce46c1652ba3aa3efc4632c09612785ca7efb964d186f4571dc054f932 not found: ID does not exist" Nov 02 14:58:15.812068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.812043 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl"] Nov 02 14:58:15.824285 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.824253 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4c2811-5575999685-4h9sl"] Nov 02 14:58:15.849711 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.849678 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm"] Nov 02 14:58:15.853445 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:15.853420 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4c2811-76d7c6647-qhghm"] Nov 02 14:58:16.089485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.089402 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" containerID="cri-o://47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532" gracePeriod=45 Nov 02 14:58:16.111422 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.111386 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" containerID="cri-o://822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab" gracePeriod=45 Nov 02 14:58:16.369165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.369135 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:58:16.382440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.382416 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425187 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-model-cache\") pod \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425238 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-home\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-home\") pod \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425287 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-dshm\") pod \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425319 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-kube-api-access-xw589\" (UniqueName: \"kubernetes.io/projected/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-kube-api-access-xw589\") pod \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425357 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tls-certs\" (UniqueName: \"kubernetes.io/secret/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tls-certs\") pod \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425474 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tmp-dir\") pod \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\" (UID: \"b347cc67-4e2f-44bd-9d7a-ab4b4774d189\") " Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425490 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-model-cache" pod "b347cc67-4e2f-44bd-9d7a-ab4b4774d189" (UID: "b347cc67-4e2f-44bd-9d7a-ab4b4774d189"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425530 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-home" pod "b347cc67-4e2f-44bd-9d7a-ab4b4774d189" (UID: "b347cc67-4e2f-44bd-9d7a-ab4b4774d189"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425735 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425749 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.426078 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.425972 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tmp-dir" pod "b347cc67-4e2f-44bd-9d7a-ab4b4774d189" (UID: "b347cc67-4e2f-44bd-9d7a-ab4b4774d189"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:16.429013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.428339 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-kube-api-access-xw589" pod "b347cc67-4e2f-44bd-9d7a-ab4b4774d189" (UID: "b347cc67-4e2f-44bd-9d7a-ab4b4774d189"). InnerVolumeSpecName "kube-api-access-xw589". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:16.429013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.428414 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-dshm" pod "b347cc67-4e2f-44bd-9d7a-ab4b4774d189" (UID: "b347cc67-4e2f-44bd-9d7a-ab4b4774d189"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:16.429013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.428559 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tls-certs" pod "b347cc67-4e2f-44bd-9d7a-ab4b4774d189" (UID: "b347cc67-4e2f-44bd-9d7a-ab4b4774d189"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:16.526094 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526053 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dc1a59-b95e-4f19-b397-c39eef342da2-kube-api-access-ft54l\" (UniqueName: \"kubernetes.io/projected/79dc1a59-b95e-4f19-b397-c39eef342da2-kube-api-access-ft54l\") pod \"79dc1a59-b95e-4f19-b397-c39eef342da2\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " Nov 02 14:58:16.526284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526111 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-dshm\") pod \"79dc1a59-b95e-4f19-b397-c39eef342da2\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " Nov 02 14:58:16.526284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526160 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-tmp-dir\") pod \"79dc1a59-b95e-4f19-b397-c39eef342da2\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " Nov 02 14:58:16.526284 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526197 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-home\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-home\") pod \"79dc1a59-b95e-4f19-b397-c39eef342da2\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " Nov 02 14:58:16.526467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526354 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79dc1a59-b95e-4f19-b397-c39eef342da2-tls-certs\" (UniqueName: \"kubernetes.io/secret/79dc1a59-b95e-4f19-b397-c39eef342da2-tls-certs\") pod \"79dc1a59-b95e-4f19-b397-c39eef342da2\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " Nov 02 14:58:16.526467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526399 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-model-cache\") pod \"79dc1a59-b95e-4f19-b397-c39eef342da2\" (UID: \"79dc1a59-b95e-4f19-b397-c39eef342da2\") " Nov 02 14:58:16.526581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526456 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-tmp-dir" pod "79dc1a59-b95e-4f19-b397-c39eef342da2" (UID: "79dc1a59-b95e-4f19-b397-c39eef342da2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:16.526581 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526521 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-home" pod "79dc1a59-b95e-4f19-b397-c39eef342da2" (UID: "79dc1a59-b95e-4f19-b397-c39eef342da2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:16.526817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526790 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-model-cache" pod "79dc1a59-b95e-4f19-b397-c39eef342da2" (UID: "79dc1a59-b95e-4f19-b397-c39eef342da2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:16.526817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526798 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.526975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526824 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.526975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526838 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw589\" (UniqueName: \"kubernetes.io/projected/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-kube-api-access-xw589\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.526975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526853 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b347cc67-4e2f-44bd-9d7a-ab4b4774d189-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.526975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526865 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.526975 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.526877 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.528287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.528261 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-dshm" pod "79dc1a59-b95e-4f19-b397-c39eef342da2" (UID: "79dc1a59-b95e-4f19-b397-c39eef342da2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:16.528514 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.528490 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79dc1a59-b95e-4f19-b397-c39eef342da2-tls-certs" pod "79dc1a59-b95e-4f19-b397-c39eef342da2" (UID: "79dc1a59-b95e-4f19-b397-c39eef342da2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:16.528743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.528723 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dc1a59-b95e-4f19-b397-c39eef342da2-kube-api-access-ft54l" pod "79dc1a59-b95e-4f19-b397-c39eef342da2" (UID: "79dc1a59-b95e-4f19-b397-c39eef342da2"). InnerVolumeSpecName "kube-api-access-ft54l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:16.627718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.627633 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79dc1a59-b95e-4f19-b397-c39eef342da2-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.627718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.627664 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.627718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.627675 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft54l\" (UniqueName: \"kubernetes.io/projected/79dc1a59-b95e-4f19-b397-c39eef342da2-kube-api-access-ft54l\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.627718 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.627683 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79dc1a59-b95e-4f19-b397-c39eef342da2-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:16.769328 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.769289 5591 generic.go:356] "Generic (PLEG): container finished" podID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerID="47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532" exitCode=0 Nov 02 14:58:16.769535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.769339 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" Nov 02 14:58:16.769535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.769382 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" event={"ID":"b347cc67-4e2f-44bd-9d7a-ab4b4774d189","Type":"ContainerDied","Data":"47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532"} Nov 02 14:58:16.769535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.769432 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz" event={"ID":"b347cc67-4e2f-44bd-9d7a-ab4b4774d189","Type":"ContainerDied","Data":"912b5821d2bddf1e2f6e68703ea3aa96a03eb1952f2ceb92a0cb3fc59d0264b4"} Nov 02 14:58:16.769535 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.769457 5591 scope.go:122] "RemoveContainer" containerID="47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532" Nov 02 14:58:16.770878 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.770850 5591 generic.go:356] "Generic (PLEG): container finished" podID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerID="822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab" exitCode=0 Nov 02 14:58:16.771005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.770886 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" Nov 02 14:58:16.771005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.770930 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" event={"ID":"79dc1a59-b95e-4f19-b397-c39eef342da2","Type":"ContainerDied","Data":"822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab"} Nov 02 14:58:16.771140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.770976 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr" event={"ID":"79dc1a59-b95e-4f19-b397-c39eef342da2","Type":"ContainerDied","Data":"75eecad4e9c5d52d2838ca788b9dbc6363c98777f8b3427babc99b3506f3e157"} Nov 02 14:58:16.790151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.789906 5591 scope.go:122] "RemoveContainer" containerID="47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532" Nov 02 14:58:16.790242 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:16.790218 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532\": container with ID starting with 47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532 not found: ID does not exist" containerID="47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532" Nov 02 14:58:16.790301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.790250 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532"} err="failed to get container status \"47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532\": rpc error: code = NotFound desc = could not find container \"47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532\": container with ID starting with 47e891484b2395a8a528476085ed03a081254db5f9f6b0c8e198f3230aa9a532 not found: ID does not exist" Nov 02 14:58:16.790301 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.790268 5591 scope.go:122] "RemoveContainer" containerID="822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab" Nov 02 14:58:16.798421 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.798400 5591 scope.go:122] "RemoveContainer" containerID="822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab" Nov 02 14:58:16.798690 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:16.798670 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab\": container with ID starting with 822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab not found: ID does not exist" containerID="822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab" Nov 02 14:58:16.798749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.798700 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab"} err="failed to get container status \"822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab\": rpc error: code = NotFound desc = could not find container \"822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab\": container with ID starting with 822a17573ce4180525891818a5d3a930bb37ab857057e009ffc08b37ac6dc0ab not found: ID does not exist" Nov 02 14:58:16.825210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.825183 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz"] Nov 02 14:58:16.837317 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.837284 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-w1-92f249-kserve-7cccbccf96-vclzz"] Nov 02 14:58:16.862339 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.862306 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr"] Nov 02 14:58:16.863786 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.863762 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/simulator-w1-92f249-kserve-5f7c55d5bd-mfwjr"] Nov 02 14:58:16.995834 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:16.995631 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:17.428382 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:17.428345 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" path="/var/lib/kubelet/pods/79dc1a59-b95e-4f19-b397-c39eef342da2/volumes" Nov 02 14:58:17.428974 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:17.428952 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" path="/var/lib/kubelet/pods/b347cc67-4e2f-44bd-9d7a-ab4b4774d189/volumes" Nov 02 14:58:17.429424 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:17.429402 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10e334b-2678-4fe5-a5da-16d9bd9890eb" path="/var/lib/kubelet/pods/c10e334b-2678-4fe5-a5da-16d9bd9890eb/volumes" Nov 02 14:58:17.430081 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:17.430053 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e254d3cf-7490-4cf8-b709-493a2ee68794" path="/var/lib/kubelet/pods/e254d3cf-7490-4cf8-b709-493a2ee68794/volumes" Nov 02 14:58:17.473877 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:17.473842 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:17.659126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:17.659092 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:18.429664 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:18.429618 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:18.481954 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:18.481916 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd"] Nov 02 14:58:18.482203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:18.482176 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" containerID="cri-o://f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9" gracePeriod=30 Nov 02 14:58:18.649462 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:18.649432 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:18.814210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:18.814132 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:18.981560 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:18.981509 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:19.146052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:19.146014 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:20.516920 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:20.516885 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:21.181977 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:21.181905 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:22.945750 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:22.945712 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:23.734573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.734528 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:23.745070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.745043 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:58:23.811032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.810933 5591 generic.go:356] "Generic (PLEG): container finished" podID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerID="f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9" exitCode=0 Nov 02 14:58:23.811032 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.811013 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" event={"ID":"fd9a4390-351a-4886-90ee-b2ca1ad9d931","Type":"ContainerDied","Data":"f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9"} Nov 02 14:58:23.811264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.811039 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" Nov 02 14:58:23.811264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.811076 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd" event={"ID":"fd9a4390-351a-4886-90ee-b2ca1ad9d931","Type":"ContainerDied","Data":"d3563655df1b606aadbba1c8cc4d0dcec0440ac7370ccbaf16ae8960281e3fbe"} Nov 02 14:58:23.811264 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.811130 5591 scope.go:122] "RemoveContainer" containerID="f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9" Nov 02 14:58:23.823098 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.823068 5591 scope.go:122] "RemoveContainer" containerID="f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9" Nov 02 14:58:23.823523 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:23.823496 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9\": container with ID starting with f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9 not found: ID does not exist" containerID="f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9" Nov 02 14:58:23.823654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.823534 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9"} err="failed to get container status \"f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9\": rpc error: code = NotFound desc = could not find container \"f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9\": container with ID starting with f9b89919a5666dfebb6e54fd28f00f4fef8bc8bc74acd706d35b659f1932fcf9 not found: ID does not exist" Nov 02 14:58:23.898140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.898101 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:23.911260 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911233 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-certs\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.911440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911300 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istiod-ca-cert\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.911440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911318 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-socket\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.911440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911341 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-envoy\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.911440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911386 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-kube-api-access-hxv9b\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-kube-api-access-hxv9b\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.911440 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911419 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-credential-socket\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.911778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911459 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-token\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-token\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.911778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911512 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-podinfo\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.911778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911562 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-certs" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:23.911778 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911575 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-data\") pod \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\" (UID: \"fd9a4390-351a-4886-90ee-b2ca1ad9d931\") " Nov 02 14:58:23.912022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911819 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-data" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:23.912022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.911827 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istiod-ca-cert" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:23.912161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.912037 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:23.912161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.912058 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:23.912161 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.912070 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:23.912324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.912181 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-credential-socket" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:23.912324 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.912252 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-socket" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:23.913719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.913693 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-envoy" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:23.914030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.914011 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-token" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:23.914137 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.914118 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-podinfo" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:58:23.914181 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:23.914123 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-kube-api-access-hxv9b" pod "fd9a4390-351a-4886-90ee-b2ca1ad9d931" (UID: "fd9a4390-351a-4886-90ee-b2ca1ad9d931"). InnerVolumeSpecName "kube-api-access-hxv9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:24.012959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.012917 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:24.012959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.012951 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:24.012959 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.012960 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxv9b\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-kube-api-access-hxv9b\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:24.013429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.012970 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9a4390-351a-4886-90ee-b2ca1ad9d931-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:24.013429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.012993 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:24.013429 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.013007 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9a4390-351a-4886-90ee-b2ca1ad9d931-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:24.120734 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.120657 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj"] Nov 02 14:58:24.136880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.136849 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4"] Nov 02 14:58:24.208261 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.208225 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd"] Nov 02 14:58:24.212104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.212078 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-wrbgd"] Nov 02 14:58:24.279229 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279192 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d"] Nov 02 14:58:24.279886 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279866 5591 cpu_manager.go:414] "containerMap: removing container" podUID="c10e334b-2678-4fe5-a5da-16d9bd9890eb" containerName="payload-processing" Nov 02 14:58:24.279930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279889 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10e334b-2678-4fe5-a5da-16d9bd9890eb" containerName="payload-processing" Nov 02 14:58:24.279930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279904 5591 cpu_manager.go:414] "containerMap: removing container" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" Nov 02 14:58:24.279930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279913 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" Nov 02 14:58:24.279930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279924 5591 cpu_manager.go:414] "containerMap: removing container" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" Nov 02 14:58:24.280084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279932 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" Nov 02 14:58:24.280084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279971 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" Nov 02 14:58:24.280084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.279976 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" Nov 02 14:58:24.280084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280007 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e254d3cf-7490-4cf8-b709-493a2ee68794" containerName="payload-pre-processing" Nov 02 14:58:24.280084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280013 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e254d3cf-7490-4cf8-b709-493a2ee68794" containerName="payload-pre-processing" Nov 02 14:58:24.280084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280026 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68" containerName="revoke-keys" Nov 02 14:58:24.280084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280031 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68" containerName="revoke-keys" Nov 02 14:58:24.280314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280113 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10e334b-2678-4fe5-a5da-16d9bd9890eb" containerName="payload-processing" Nov 02 14:58:24.280314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280123 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" containerName="istio-proxy" Nov 02 14:58:24.280314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280129 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="79dc1a59-b95e-4f19-b397-c39eef342da2" containerName="main" Nov 02 14:58:24.280314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280139 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b347cc67-4e2f-44bd-9d7a-ab4b4774d189" containerName="main" Nov 02 14:58:24.280314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280148 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e254d3cf-7490-4cf8-b709-493a2ee68794" containerName="payload-pre-processing" Nov 02 14:58:24.280314 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.280157 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68" containerName="revoke-keys" Nov 02 14:58:24.284605 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.284586 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.305837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.305810 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d"] Nov 02 14:58:24.306048 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.305978 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ad42e0-openshift-default-dockercfg-gmvz9\"" Nov 02 14:58:24.320118 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.320086 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:24.416373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416340 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.416543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416411 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.416543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416496 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.416543 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416525 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.416743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416554 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tmfq\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-kube-api-access-7tmfq\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.416743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416607 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.416743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416712 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360d3357-25a5-4ea0-9eaa-2edc674186d2-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.416890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416775 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.416890 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.416816 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.517686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517652 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.517686 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517691 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517735 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517768 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517789 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517813 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tmfq\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-kube-api-access-7tmfq\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517841 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517882 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360d3357-25a5-4ea0-9eaa-2edc674186d2-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518052 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.517919 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.518141 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.518234 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.518323 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518528 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.518492 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.518727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.518704 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360d3357-25a5-4ea0-9eaa-2edc674186d2-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.520223 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.520199 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.520381 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.520361 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.540788 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.540757 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.540930 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.540910 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tmfq\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-kube-api-access-7tmfq\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.597158 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.597125 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:24.855459 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.855359 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:24.924149 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.923869 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d"] Nov 02 14:58:24.926974 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:58:24.926943 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360d3357_25a5_4ea0_9eaa_2edc674186d2.slice/crio-a8f2566d775e394edd8d9b0050ff122b7a65491edf94c435ec96bf06cc6862b4 WatchSource:0}: Error finding container a8f2566d775e394edd8d9b0050ff122b7a65491edf94c435ec96bf06cc6862b4: Status 404 returned error can't find the container with id a8f2566d775e394edd8d9b0050ff122b7a65491edf94c435ec96bf06cc6862b4 Nov 02 14:58:24.929216 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.929183 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:58:24.929291 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.929250 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:58:24.929291 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:24.929278 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:58:25.428654 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:25.428616 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9a4390-351a-4886-90ee-b2ca1ad9d931" path="/var/lib/kubelet/pods/fd9a4390-351a-4886-90ee-b2ca1ad9d931/volumes" Nov 02 14:58:25.488247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:25.488211 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:25.648347 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:25.648308 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:25.824446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:25.824330 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" event={"ID":"360d3357-25a5-4ea0-9eaa-2edc674186d2","Type":"ContainerStarted","Data":"189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48"} Nov 02 14:58:25.824446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:25.824388 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" event={"ID":"360d3357-25a5-4ea0-9eaa-2edc674186d2","Type":"ContainerStarted","Data":"a8f2566d775e394edd8d9b0050ff122b7a65491edf94c435ec96bf06cc6862b4"} Nov 02 14:58:25.885318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:25.885260 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podStartSLOduration=1.885245389 podStartE2EDuration="1.885245389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:25.883517677 +0000 UTC m=+4454.609139447" watchObservedRunningTime="2026-11-02 14:58:25.885245389 +0000 UTC m=+4454.610867212" Nov 02 14:58:26.360709 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:26.360663 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:26.564449 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:26.563974 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:26.598013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:26.597967 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:26.599485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:26.599456 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 14:58:26.599628 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:26.599507 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 14:58:27.171200 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:27.171163 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg"] Nov 02 14:58:27.171447 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:27.171418 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" containerID="cri-o://8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc" gracePeriod=30 Nov 02 14:58:27.598163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:27.598071 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 14:58:27.598163 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:27.598131 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 14:58:28.184586 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:28.184543 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:28.598005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:28.597894 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 14:58:28.598005 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:28.597953 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 14:58:29.051553 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:29.051513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:29.212000 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:29.211936 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:29.598340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:29.598303 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 14:58:29.598541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:29.598356 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 14:58:30.598333 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:30.598302 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 14:58:30.598731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:30.598354 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 14:58:30.666287 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:30.666232 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:31.080905 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:31.080843 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:58:31.519294 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:31.519261 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:31.598375 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:31.598333 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body= Nov 02 14:58:31.598765 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:31.598393 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" Nov 02 14:58:31.831002 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:31.830882 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:31.993189 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:31.993146 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:32.156373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.156303 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:32.434208 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.434184 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:58:32.516777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.513223 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:32.622792 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.622751 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-istio-token\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-istio-token\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.622801 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-data\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.622825 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11918adc-cce0-49a6-9f2b-30ea03866457-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11918adc-cce0-49a6-9f2b-30ea03866457-istiod-ca-cert\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.622855 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11918adc-cce0-49a6-9f2b-30ea03866457-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11918adc-cce0-49a6-9f2b-30ea03866457-istio-podinfo\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.622890 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-credential-socket\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.622921 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-socket\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623038 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-kube-api-access-f7fkp\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-kube-api-access-f7fkp\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623088 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-envoy\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623135 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-data" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:32.623316 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623147 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-certs\") pod \"11918adc-cce0-49a6-9f2b-30ea03866457\" (UID: \"11918adc-cce0-49a6-9f2b-30ea03866457\") " Nov 02 14:58:32.623842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623344 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-socket" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:32.623842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623480 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-credential-socket" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:32.623842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623560 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-certs" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:32.623842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623675 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.623842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623693 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.623842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623706 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.623842 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.623721 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.624273 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.624081 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11918adc-cce0-49a6-9f2b-30ea03866457-istiod-ca-cert" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:32.625331 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.625293 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-istio-token" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:32.625698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.625670 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11918adc-cce0-49a6-9f2b-30ea03866457-istio-podinfo" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:58:32.625830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.625790 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-kube-api-access-f7fkp" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "kube-api-access-f7fkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:32.625830 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.625810 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-envoy" pod "11918adc-cce0-49a6-9f2b-30ea03866457" (UID: "11918adc-cce0-49a6-9f2b-30ea03866457"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:32.724568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.724478 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.724568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.724509 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11918adc-cce0-49a6-9f2b-30ea03866457-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.724568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.724518 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11918adc-cce0-49a6-9f2b-30ea03866457-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.724568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.724527 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7fkp\" (UniqueName: \"kubernetes.io/projected/11918adc-cce0-49a6-9f2b-30ea03866457-kube-api-access-f7fkp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.724568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.724537 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11918adc-cce0-49a6-9f2b-30ea03866457-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:32.895577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.895539 5591 generic.go:356] "Generic (PLEG): container finished" podID="11918adc-cce0-49a6-9f2b-30ea03866457" containerID="8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc" exitCode=0 Nov 02 14:58:32.895806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.895598 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" Nov 02 14:58:32.895806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.895600 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" event={"ID":"11918adc-cce0-49a6-9f2b-30ea03866457","Type":"ContainerDied","Data":"8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc"} Nov 02 14:58:32.895806 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.895772 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg" event={"ID":"11918adc-cce0-49a6-9f2b-30ea03866457","Type":"ContainerDied","Data":"6343ee9e5e94e55cfd52378bd43d777d955251f57628cca7987d199a79522f74"} Nov 02 14:58:32.896037 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.895810 5591 scope.go:122] "RemoveContainer" containerID="8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc" Nov 02 14:58:32.906793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.906772 5591 scope.go:122] "RemoveContainer" containerID="8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc" Nov 02 14:58:32.907053 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:32.907032 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc\": container with ID starting with 8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc not found: ID does not exist" containerID="8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc" Nov 02 14:58:32.907135 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.907061 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc"} err="failed to get container status \"8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc\": rpc error: code = NotFound desc = could not find container \"8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc\": container with ID starting with 8550fa267cf46dd4af9fe6c04b2ed599eb3dbb781d9573aa02a7840d43d268dc not found: ID does not exist" Nov 02 14:58:32.949407 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.949363 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg"] Nov 02 14:58:32.961126 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.961089 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:32.962129 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:32.962099 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4rwmg"] Nov 02 14:58:33.122481 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:33.122393 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:33.430128 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:33.430093 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" path="/var/lib/kubelet/pods/11918adc-cce0-49a6-9f2b-30ea03866457/volumes" Nov 02 14:58:33.639211 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:33.639166 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:58:33.639783 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:33.639244 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:58:33.874802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:33.874710 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:33.875140 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:33.875114 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:33.876097 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:33.876076 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:36.138712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:36.138675 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:58:38.628497 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:38.628456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:38.953787 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:38.953643 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:39.141942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.141898 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" containerID="cri-o://d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708" gracePeriod=45 Nov 02 14:58:39.155995 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.155947 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" containerID="cri-o://22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca" gracePeriod=45 Nov 02 14:58:39.313601 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.313511 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:39.412485 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.412456 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:58:39.440072 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.440048 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:58:39.488203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488116 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tls-certs\" (UniqueName: \"kubernetes.io/secret/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tls-certs\") pod \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " Nov 02 14:58:39.488203 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488167 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-dshm\") pod \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " Nov 02 14:58:39.488439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488286 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a01bbb4-da32-470c-8ba4-54ac28a79c19-kube-api-access-ldlkz\" (UniqueName: \"kubernetes.io/projected/2a01bbb4-da32-470c-8ba4-54ac28a79c19-kube-api-access-ldlkz\") pod \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " Nov 02 14:58:39.488439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488343 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tmp-dir\") pod \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " Nov 02 14:58:39.488439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488385 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tls-certs\") pod \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " Nov 02 14:58:39.488439 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488426 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-kube-api-access-98xxr\" (UniqueName: \"kubernetes.io/projected/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-kube-api-access-98xxr\") pod \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " Nov 02 14:58:39.488677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488460 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-model-cache\") pod \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " Nov 02 14:58:39.488677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488519 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-model-cache\") pod \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " Nov 02 14:58:39.488677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488546 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-home\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-home\") pod \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " Nov 02 14:58:39.488677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488573 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-dshm\") pod \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " Nov 02 14:58:39.488677 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488601 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-home\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-home\") pod \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\" (UID: \"2a01bbb4-da32-470c-8ba4-54ac28a79c19\") " Nov 02 14:58:39.489009 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.488702 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tmp-dir\") pod \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\" (UID: \"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179\") " Nov 02 14:58:39.490615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.489338 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tmp-dir" pod "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" (UID: "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.490615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.489472 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tmp-dir" pod "2a01bbb4-da32-470c-8ba4-54ac28a79c19" (UID: "2a01bbb4-da32-470c-8ba4-54ac28a79c19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.490615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.489565 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-home" pod "2a01bbb4-da32-470c-8ba4-54ac28a79c19" (UID: "2a01bbb4-da32-470c-8ba4-54ac28a79c19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.490615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.489711 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-home" pod "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" (UID: "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.490615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.489757 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-model-cache" pod "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" (UID: "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.490615 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.489876 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-model-cache" pod "2a01bbb4-da32-470c-8ba4-54ac28a79c19" (UID: "2a01bbb4-da32-470c-8ba4-54ac28a79c19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.491248 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.491224 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-dshm" pod "2a01bbb4-da32-470c-8ba4-54ac28a79c19" (UID: "2a01bbb4-da32-470c-8ba4-54ac28a79c19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.492472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.492425 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tls-certs" pod "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" (UID: "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:39.492472 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.492446 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-dshm" pod "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" (UID: "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.492671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.492541 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-kube-api-access-98xxr" pod "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" (UID: "01723d00-e2c0-4ab1-bbee-ff8a7d6ac179"). InnerVolumeSpecName "kube-api-access-98xxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:39.492671 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.492575 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a01bbb4-da32-470c-8ba4-54ac28a79c19-kube-api-access-ldlkz" pod "2a01bbb4-da32-470c-8ba4-54ac28a79c19" (UID: "2a01bbb4-da32-470c-8ba4-54ac28a79c19"). InnerVolumeSpecName "kube-api-access-ldlkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:39.492880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.492862 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tls-certs" pod "2a01bbb4-da32-470c-8ba4-54ac28a79c19" (UID: "2a01bbb4-da32-470c-8ba4-54ac28a79c19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:39.589896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.589852 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.589896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.589893 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.589896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.589905 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.589917 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.589954 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.589967 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.589992 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.590007 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldlkz\" (UniqueName: \"kubernetes.io/projected/2a01bbb4-da32-470c-8ba4-54ac28a79c19-kube-api-access-ldlkz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.590023 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.590038 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01bbb4-da32-470c-8ba4-54ac28a79c19-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.590053 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xxr\" (UniqueName: \"kubernetes.io/projected/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-kube-api-access-98xxr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.590242 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.590066 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:39.604527 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.604488 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:39.940899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.940371 5591 generic.go:356] "Generic (PLEG): container finished" podID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerID="d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708" exitCode=0 Nov 02 14:58:39.940899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.940574 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" event={"ID":"2a01bbb4-da32-470c-8ba4-54ac28a79c19","Type":"ContainerDied","Data":"d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708"} Nov 02 14:58:39.940899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.940613 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" event={"ID":"2a01bbb4-da32-470c-8ba4-54ac28a79c19","Type":"ContainerDied","Data":"5ddcce4b9cd5e983cf92cf1bf0684b8e730fdbab46bf1ade0d17fb6272877dd8"} Nov 02 14:58:39.940899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.940634 5591 scope.go:122] "RemoveContainer" containerID="d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708" Nov 02 14:58:39.940899 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.940766 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj" Nov 02 14:58:39.942809 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.942775 5591 generic.go:356] "Generic (PLEG): container finished" podID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerID="22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca" exitCode=0 Nov 02 14:58:39.942958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.942812 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" event={"ID":"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179","Type":"ContainerDied","Data":"22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca"} Nov 02 14:58:39.942958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.942830 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" Nov 02 14:58:39.942958 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.942851 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" event={"ID":"01723d00-e2c0-4ab1-bbee-ff8a7d6ac179","Type":"ContainerDied","Data":"bab878d164fb18126c732c96ce20dc869b3806c0fe5316bdbf2dd778fea5c657"} Nov 02 14:58:39.960022 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.959973 5591 scope.go:122] "RemoveContainer" containerID="d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708" Nov 02 14:58:39.960391 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:39.960368 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708\": container with ID starting with d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708 not found: ID does not exist" containerID="d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708" Nov 02 14:58:39.960487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.960404 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708"} err="failed to get container status \"d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708\": rpc error: code = NotFound desc = could not find container \"d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708\": container with ID starting with d107edab6d5d0c7121c00fd47e511cf5093cf68625fed9ad6a8964834007b708 not found: ID does not exist" Nov 02 14:58:39.960487 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.960428 5591 scope.go:122] "RemoveContainer" containerID="22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca" Nov 02 14:58:39.969450 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.969423 5591 scope.go:122] "RemoveContainer" containerID="22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca" Nov 02 14:58:39.969749 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:39.969727 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca\": container with ID starting with 22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca not found: ID does not exist" containerID="22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca" Nov 02 14:58:39.969820 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.969760 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca"} err="failed to get container status \"22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca\": rpc error: code = NotFound desc = could not find container \"22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca\": container with ID starting with 22ff00df9b57e00f76c8084a9ba99a0ffa92d52ea3ee4a8677e16ba2f9e37aca not found: ID does not exist" Nov 02 14:58:39.991865 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.991834 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4"] Nov 02 14:58:39.996639 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:39.996612 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:40.003030 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:40.002995 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4"] Nov 02 14:58:40.032231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:40.032193 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj"] Nov 02 14:58:40.036247 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:40.036213 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-2-w2-0969b1-kserve-cfdf985c5-mz9pj"] Nov 02 14:58:40.388603 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:40.388550 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/premium-w2-0969b1-kserve-76ccc6cb7c-vzlz4" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:58:40.777278 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:40.777193 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:41.429027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:41.428974 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" path="/var/lib/kubelet/pods/01723d00-e2c0-4ab1-bbee-ff8a7d6ac179/volumes" Nov 02 14:58:41.429562 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:41.429547 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" path="/var/lib/kubelet/pods/2a01bbb4-da32-470c-8ba4-54ac28a79c19/volumes" Nov 02 14:58:41.915183 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:41.915146 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:42.320743 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:42.320663 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:42.788619 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:42.788567 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:43.989096 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:43.989059 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp"] Nov 02 14:58:43.989513 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:43.989308 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" podUID="34c61935-488e-4e61-85d4-618c252f458f" containerName="maas-api" containerID="cri-o://6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d" gracePeriod=30 Nov 02 14:58:44.273070 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.273033 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 14:58:44.328772 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.328740 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:44.338578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.338542 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c61935-488e-4e61-85d4-618c252f458f-kube-api-access-2xtx8\" (UniqueName: \"kubernetes.io/projected/34c61935-488e-4e61-85d4-618c252f458f-kube-api-access-2xtx8\") pod \"34c61935-488e-4e61-85d4-618c252f458f\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " Nov 02 14:58:44.338719 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.338651 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c61935-488e-4e61-85d4-618c252f458f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c61935-488e-4e61-85d4-618c252f458f-openshift-service-ca\") pod \"34c61935-488e-4e61-85d4-618c252f458f\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " Nov 02 14:58:44.338767 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.338721 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls\") pod \"34c61935-488e-4e61-85d4-618c252f458f\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " Nov 02 14:58:44.338813 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.338764 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-metrics-certs\") pod \"34c61935-488e-4e61-85d4-618c252f458f\" (UID: \"34c61935-488e-4e61-85d4-618c252f458f\") " Nov 02 14:58:44.339074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.339045 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c61935-488e-4e61-85d4-618c252f458f-openshift-service-ca" pod "34c61935-488e-4e61-85d4-618c252f458f" (UID: "34c61935-488e-4e61-85d4-618c252f458f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:44.339190 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.339120 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34c61935-488e-4e61-85d4-618c252f458f-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:44.340843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.340804 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c61935-488e-4e61-85d4-618c252f458f-kube-api-access-2xtx8" pod "34c61935-488e-4e61-85d4-618c252f458f" (UID: "34c61935-488e-4e61-85d4-618c252f458f"). InnerVolumeSpecName "kube-api-access-2xtx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:44.340934 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.340838 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-metrics-certs" pod "34c61935-488e-4e61-85d4-618c252f458f" (UID: "34c61935-488e-4e61-85d4-618c252f458f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:44.341195 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.341175 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls" pod "34c61935-488e-4e61-85d4-618c252f458f" (UID: "34c61935-488e-4e61-85d4-618c252f458f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:44.440315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.440237 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:44.440315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.440275 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c61935-488e-4e61-85d4-618c252f458f-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:44.440315 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.440290 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtx8\" (UniqueName: \"kubernetes.io/projected/34c61935-488e-4e61-85d4-618c252f458f-kube-api-access-2xtx8\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:44.558150 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.558062 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:44.925749 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.925713 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d"] Nov 02 14:58:44.925948 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.925923 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" containerID="cri-o://189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48" gracePeriod=30 Nov 02 14:58:44.970999 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.970942 5591 generic.go:356] "Generic (PLEG): container finished" podID="34c61935-488e-4e61-85d4-618c252f458f" containerID="6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d" exitCode=0 Nov 02 14:58:44.971263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.971008 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" Nov 02 14:58:44.971263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.971011 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" event={"ID":"34c61935-488e-4e61-85d4-618c252f458f","Type":"ContainerDied","Data":"6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d"} Nov 02 14:58:44.971263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.971144 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp" event={"ID":"34c61935-488e-4e61-85d4-618c252f458f","Type":"ContainerDied","Data":"7a3b84afa21b2b7f5f95bc5dd9d78c883466d95adb5ed1fabfc7a5de29dc897c"} Nov 02 14:58:44.971263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.971179 5591 scope.go:122] "RemoveContainer" containerID="6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d" Nov 02 14:58:44.982896 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.982874 5591 scope.go:122] "RemoveContainer" containerID="6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d" Nov 02 14:58:44.983274 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:44.983248 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d\": container with ID starting with 6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d not found: ID does not exist" containerID="6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d" Nov 02 14:58:44.983365 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:44.983282 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d"} err="failed to get container status \"6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d\": rpc error: code = NotFound desc = could not find container \"6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d\": container with ID starting with 6b42e7d5a7a00954d5ef03554569dce90da5aa43eabe2aa394bad89d6335732d not found: ID does not exist" Nov 02 14:58:45.025209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:45.025168 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:45.031952 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:45.031914 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp"] Nov 02 14:58:45.032978 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:45.032948 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0969b1-695bb6ffb8-w7dsp"] Nov 02 14:58:45.429080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:45.429046 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c61935-488e-4e61-85d4-618c252f458f" path="/var/lib/kubelet/pods/34c61935-488e-4e61-85d4-618c252f458f/volumes" Nov 02 14:58:46.982965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:46.982920 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:47.177622 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:47.177572 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:48.311061 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:48.309368 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:58:48.878093 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:48.878047 5591 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 02 14:58:48.878288 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:48.878106 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:58:49.958209 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:49.958176 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:50.124965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.124928 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:50.178415 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.178390 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:50.309397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309303 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-socket\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309397 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309354 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-envoy\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309395 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360d3357-25a5-4ea0-9eaa-2edc674186d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360d3357-25a5-4ea0-9eaa-2edc674186d2-istiod-ca-cert\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309435 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-kube-api-access-7tmfq\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-kube-api-access-7tmfq\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309459 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-certs\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309499 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-data\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309529 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-credential-socket\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309582 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-token\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-token\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309648 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309618 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-socket" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:50.309893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309632 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-podinfo\") pod \"360d3357-25a5-4ea0-9eaa-2edc674186d2\" (UID: \"360d3357-25a5-4ea0-9eaa-2edc674186d2\") " Nov 02 14:58:50.309893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309835 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360d3357-25a5-4ea0-9eaa-2edc674186d2-istiod-ca-cert" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:50.310020 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.309976 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/360d3357-25a5-4ea0-9eaa-2edc674186d2-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:50.310079 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.310023 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:50.311254 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.311203 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-certs" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:50.311412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.311298 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-credential-socket" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:50.311412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.311317 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-data" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:50.311763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.311737 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-envoy" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:50.313143 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.313119 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-kube-api-access-7tmfq" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "kube-api-access-7tmfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:50.313255 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.313176 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-podinfo" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:58:50.313363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.313340 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-token" pod "360d3357-25a5-4ea0-9eaa-2edc674186d2" (UID: "360d3357-25a5-4ea0-9eaa-2edc674186d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:50.411232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.411196 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:50.411232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.411225 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:50.411232 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.411234 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:50.411541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.411243 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tmfq\" (UniqueName: \"kubernetes.io/projected/360d3357-25a5-4ea0-9eaa-2edc674186d2-kube-api-access-7tmfq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:50.411541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.411254 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:50.411541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.411262 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:50.411541 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:50.411272 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/360d3357-25a5-4ea0-9eaa-2edc674186d2-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:58:51.006475 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.006439 5591 generic.go:356] "Generic (PLEG): container finished" podID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerID="189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48" exitCode=0 Nov 02 14:58:51.006942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.006493 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" Nov 02 14:58:51.006942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.006511 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" event={"ID":"360d3357-25a5-4ea0-9eaa-2edc674186d2","Type":"ContainerDied","Data":"189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48"} Nov 02 14:58:51.006942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.006557 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d" event={"ID":"360d3357-25a5-4ea0-9eaa-2edc674186d2","Type":"ContainerDied","Data":"a8f2566d775e394edd8d9b0050ff122b7a65491edf94c435ec96bf06cc6862b4"} Nov 02 14:58:51.006942 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.006578 5591 scope.go:122] "RemoveContainer" containerID="189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48" Nov 02 14:58:51.015777 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.015754 5591 scope.go:122] "RemoveContainer" containerID="189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48" Nov 02 14:58:51.016069 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:58:51.016043 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48\": container with ID starting with 189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48 not found: ID does not exist" containerID="189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48" Nov 02 14:58:51.016159 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.016075 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48"} err="failed to get container status \"189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48\": rpc error: code = NotFound desc = could not find container \"189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48\": container with ID starting with 189fcaa11ba012ba1395228e71c5ac6e30614a01fbd08a37cabc23a13edc3b48 not found: ID does not exist" Nov 02 14:58:51.068318 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.068282 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d"] Nov 02 14:58:51.079814 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.079784 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-whh5d"] Nov 02 14:58:51.429579 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:51.429546 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" path="/var/lib/kubelet/pods/360d3357-25a5-4ea0-9eaa-2edc674186d2/volumes" Nov 02 14:58:53.460087 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460053 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2"] Nov 02 14:58:53.460493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460408 5591 cpu_manager.go:414] "containerMap: removing container" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" Nov 02 14:58:53.460493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460424 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" Nov 02 14:58:53.460493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460437 5591 cpu_manager.go:414] "containerMap: removing container" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" Nov 02 14:58:53.460493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460443 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" Nov 02 14:58:53.460493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460465 5591 cpu_manager.go:414] "containerMap: removing container" podUID="34c61935-488e-4e61-85d4-618c252f458f" containerName="maas-api" Nov 02 14:58:53.460493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460470 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c61935-488e-4e61-85d4-618c252f458f" containerName="maas-api" Nov 02 14:58:53.460493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460485 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" Nov 02 14:58:53.460493 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460490 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" Nov 02 14:58:53.460757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460499 5591 cpu_manager.go:414] "containerMap: removing container" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" Nov 02 14:58:53.460757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460503 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" Nov 02 14:58:53.460757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460555 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c61935-488e-4e61-85d4-618c252f458f" containerName="maas-api" Nov 02 14:58:53.460757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460563 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a01bbb4-da32-470c-8ba4-54ac28a79c19" containerName="main" Nov 02 14:58:53.460757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460572 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="360d3357-25a5-4ea0-9eaa-2edc674186d2" containerName="istio-proxy" Nov 02 14:58:53.460757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460579 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="11918adc-cce0-49a6-9f2b-30ea03866457" containerName="istio-proxy" Nov 02 14:58:53.460757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.460585 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="01723d00-e2c0-4ab1-bbee-ff8a7d6ac179" containerName="main" Nov 02 14:58:53.464840 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.464817 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.479465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.479443 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-0969b1-openshift-default-dockercfg-8rlbj\"" Nov 02 14:58:53.492892 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.492868 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2"] Nov 02 14:58:53.539337 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539304 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.539520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539347 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380e3faf-8127-4aee-8b1f-695cc27c1d71-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.539520 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539468 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.539670 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539526 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.539744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539670 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.539744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539717 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.539744 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539740 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.539883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539809 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbvt\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-kube-api-access-vnbvt\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.539883 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.539851 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641117 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641074 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641125 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380e3faf-8127-4aee-8b1f-695cc27c1d71-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641159 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641197 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641257 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641300 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641290 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641319 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641347 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbvt\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-kube-api-access-vnbvt\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641561 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641373 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641573 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641727 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641621 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641773 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.641844 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.641828 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.642105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.642082 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380e3faf-8127-4aee-8b1f-695cc27c1d71-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.643573 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.643553 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.644105 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.644084 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.664510 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.664485 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.664640 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.664544 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbvt\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-kube-api-access-vnbvt\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:53.777819 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:53.777735 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:54.093043 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:54.092943 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2"] Nov 02 14:58:54.096455 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:58:54.096426 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380e3faf_8127_4aee_8b1f_695cc27c1d71.slice/crio-9166e8e37b40787a61204efa6f604ab15025187de0d950275078f6f751fc7267 WatchSource:0}: Error finding container 9166e8e37b40787a61204efa6f604ab15025187de0d950275078f6f751fc7267: Status 404 returned error can't find the container with id 9166e8e37b40787a61204efa6f604ab15025187de0d950275078f6f751fc7267 Nov 02 14:58:54.098517 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:54.098484 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:58:54.098593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:54.098552 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:58:54.098593 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:54.098580 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 14:58:54.799210 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:54.799166 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:54.972235 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:54.972197 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:55.029213 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:55.029174 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" event={"ID":"380e3faf-8127-4aee-8b1f-695cc27c1d71","Type":"ContainerStarted","Data":"76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5"} Nov 02 14:58:55.029446 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:55.029274 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" event={"ID":"380e3faf-8127-4aee-8b1f-695cc27c1d71","Type":"ContainerStarted","Data":"9166e8e37b40787a61204efa6f604ab15025187de0d950275078f6f751fc7267"} Nov 02 14:58:55.086676 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:55.086566 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" podStartSLOduration=2.086552686 podStartE2EDuration="2.086552686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:55.085644711 +0000 UTC m=+4483.811266481" watchObservedRunningTime="2026-11-02 14:58:55.086552686 +0000 UTC m=+4483.812174457" Nov 02 14:58:55.783829 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:55.783784 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:58:55.784156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:55.784124 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Nov 02 14:58:55.784231 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:55.784184 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Nov 02 14:58:56.778808 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:56.778772 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Nov 02 14:58:56.779257 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:56.778828 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Nov 02 14:58:57.160518 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:57.160461 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:57.778658 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:57.778617 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Nov 02 14:58:57.779073 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:57.778688 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Nov 02 14:58:58.573793 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:58.573749 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:58.778655 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:58.778615 5591 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body= Nov 02 14:58:58.778843 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:58.778672 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" Nov 02 14:58:59.158013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:59.146490 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2"] Nov 02 14:58:59.158880 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:59.158832 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerName="istio-proxy" containerID="cri-o://76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5" gracePeriod=30 Nov 02 14:58:59.218156 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:58:59.218119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:00.510902 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:00.510856 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:00.700904 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:00.700860 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:01.073566 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:59:01.073521 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:59:01.195780 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:01.195747 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sj286_d529065b-33fe-4288-b543-a51b5de16f88/manager/0.log" Nov 02 14:59:01.972399 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:01.972367 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4zfwv_1c4e7da5-37b7-45bd-9840-6c87ab416429/limitador/0.log" Nov 02 14:59:02.167473 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:02.167439 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-sxzjh_c7a64f22-0110-479d-900e-b27ee83b6985/manager/0.log" Nov 02 14:59:02.866597 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:02.866560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:03.028148 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:03.028108 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:03.138893 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:03.138861 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b7f5f45bd-8zr8v_a658ad5a-9b96-42c1-af92-c22f12558516/manager/0.log" Nov 02 14:59:05.614891 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.614860 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:59:05.658263 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658233 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-podinfo\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658267 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-certs\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658300 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380e3faf-8127-4aee-8b1f-695cc27c1d71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380e3faf-8127-4aee-8b1f-695cc27c1d71-istiod-ca-cert\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658329 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-data\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658349 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-envoy\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658390 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-credential-socket\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658430 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658416 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-socket\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658445 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-token\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-token\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658477 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-kube-api-access-vnbvt\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-kube-api-access-vnbvt\") pod \"380e3faf-8127-4aee-8b1f-695cc27c1d71\" (UID: \"380e3faf-8127-4aee-8b1f-695cc27c1d71\") " Nov 02 14:59:05.658785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658545 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-certs" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:59:05.658785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658665 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-data" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:59:05.658785 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658715 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-socket" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:59:05.659080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658824 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:05.659080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658844 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:05.659080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658858 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:05.659080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658868 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380e3faf-8127-4aee-8b1f-695cc27c1d71-istiod-ca-cert" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:59:05.659080 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.658872 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-credential-socket" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:59:05.660577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.660554 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-podinfo" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:59:05.660716 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.660696 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-kube-api-access-vnbvt" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "kube-api-access-vnbvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:59:05.660766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.660712 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-envoy" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:59:05.660766 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.660729 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-token" pod "380e3faf-8127-4aee-8b1f-695cc27c1d71" (UID: "380e3faf-8127-4aee-8b1f-695cc27c1d71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:59:05.760165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.760082 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:05.760165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.760109 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380e3faf-8127-4aee-8b1f-695cc27c1d71-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:05.760165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.760118 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:05.760165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.760127 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbvt\" (UniqueName: \"kubernetes.io/projected/380e3faf-8127-4aee-8b1f-695cc27c1d71-kube-api-access-vnbvt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:05.760165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.760135 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380e3faf-8127-4aee-8b1f-695cc27c1d71-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:05.760165 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:05.760143 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380e3faf-8127-4aee-8b1f-695cc27c1d71-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:06.096775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.096683 5591 generic.go:356] "Generic (PLEG): container finished" podID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerID="76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5" exitCode=0 Nov 02 14:59:06.096775 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.096726 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" Nov 02 14:59:06.096976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.096728 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" event={"ID":"380e3faf-8127-4aee-8b1f-695cc27c1d71","Type":"ContainerDied","Data":"76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5"} Nov 02 14:59:06.096976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.096865 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2" event={"ID":"380e3faf-8127-4aee-8b1f-695cc27c1d71","Type":"ContainerDied","Data":"9166e8e37b40787a61204efa6f604ab15025187de0d950275078f6f751fc7267"} Nov 02 14:59:06.096976 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.096897 5591 scope.go:122] "RemoveContainer" containerID="76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5" Nov 02 14:59:06.105965 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.105944 5591 scope.go:122] "RemoveContainer" containerID="76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5" Nov 02 14:59:06.106258 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:59:06.106237 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5\": container with ID starting with 76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5 not found: ID does not exist" containerID="76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5" Nov 02 14:59:06.106309 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.106267 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5"} err="failed to get container status \"76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5\": rpc error: code = NotFound desc = could not find container \"76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5\": container with ID starting with 76c1b84e8fab7f6e372013ce9fefbd5c5b8f20ce668acda3da2c8e99579f49e5 not found: ID does not exist" Nov 02 14:59:06.147698 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.147668 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2"] Nov 02 14:59:06.159798 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:06.159768 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-2qtr2"] Nov 02 14:59:07.429186 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:07.429151 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" path="/var/lib/kubelet/pods/380e3faf-8127-4aee-8b1f-695cc27c1d71/volumes" Nov 02 14:59:12.323714 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:12.323688 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:59:12.347239 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:12.347211 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 14:59:21.338084 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:21.338042 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:21.894757 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:21.894725 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:22.061272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:22.061227 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:24.688846 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:24.688768 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:24.859258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:24.859226 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:26.087320 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.087285 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc"] Nov 02 14:59:26.087731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.087678 5591 cpu_manager.go:414] "containerMap: removing container" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerName="istio-proxy" Nov 02 14:59:26.087731 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.087690 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerName="istio-proxy" Nov 02 14:59:26.087810 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.087750 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="380e3faf-8127-4aee-8b1f-695cc27c1d71" containerName="istio-proxy" Nov 02 14:59:26.091363 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.091335 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.118739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.118703 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 14:59:26.118739 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.118718 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fcad4d9a-serving-cert\"" Nov 02 14:59:26.118967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.118781 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 14:59:26.118967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.118864 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 14:59:26.118967 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.118887 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 14:59:26.120617 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.120593 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc"] Nov 02 14:59:26.140179 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.140143 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-metrics-certs\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.140346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.140189 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk9l6\" (UniqueName: \"kubernetes.io/projected/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-kube-api-access-mk9l6\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.140346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.140261 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-openshift-service-ca\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.140346 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.140319 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-maas-api-tls\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.233013 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.232943 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:26.241781 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.241752 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-metrics-certs\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.241932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.241793 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk9l6\" (UniqueName: \"kubernetes.io/projected/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-kube-api-access-mk9l6\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.241932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.241819 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-openshift-service-ca\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.241932 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.241861 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-maas-api-tls\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.242568 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.242544 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-openshift-service-ca\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.244252 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.244228 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-maas-api-tls\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.244335 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.244277 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-metrics-certs\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.268068 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.268032 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk9l6\" (UniqueName: \"kubernetes.io/projected/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-kube-api-access-mk9l6\") pod \"maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.400645 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.400611 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:26.407570 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.407543 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:26.582290 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.582253 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:26.708755 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.708665 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc"] Nov 02 14:59:26.712528 ip-10-0-140-236 kubenswrapper[5591]: W1102 14:59:26.712491 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c415b9_27f5_4f3c_85d5_a7cba8c71ccd.slice/crio-40a66f35711070ef31c4733e4a5e8b7cae88c228d2694e28987f49dbb295b44d WatchSource:0}: Error finding container 40a66f35711070ef31c4733e4a5e8b7cae88c228d2694e28987f49dbb295b44d: Status 404 returned error can't find the container with id 40a66f35711070ef31c4733e4a5e8b7cae88c228d2694e28987f49dbb295b44d Nov 02 14:59:26.713916 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:26.713899 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:59:27.199057 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:27.199017 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" event={"ID":"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd","Type":"ContainerStarted","Data":"6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b"} Nov 02 14:59:27.199465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:27.199084 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" event={"ID":"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd","Type":"ContainerStarted","Data":"40a66f35711070ef31c4733e4a5e8b7cae88c228d2694e28987f49dbb295b44d"} Nov 02 14:59:27.199465 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:27.199238 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:27.242258 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:27.242203 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" podStartSLOduration=0.853719928 podStartE2EDuration="1.242185763s" totalImagesPullingTime="388.465835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:59:27.241765799 +0000 UTC m=+4515.967387584" watchObservedRunningTime="2026-11-02 14:59:27.242185763 +0000 UTC m=+4515.967807532" Nov 02 14:59:28.056340 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:28.056305 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:29.158864 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:29.158813 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:29.323578 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:29.323531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:29.822244 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:29.822214 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:29.936659 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:29.936617 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:59:31.073464 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:59:31.073424 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 14:59:31.483327 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:31.483291 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:33.151024 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:33.150992 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:33.217272 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:33.217239 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:34.829270 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:34.829233 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:36.588366 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:36.588326 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:38.151529 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:38.151497 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:38.311763 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:38.311731 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:39.864292 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:39.864255 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:41.663224 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:41.663191 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:43.353027 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:43.352963 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:45.055441 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:45.055403 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:46.651592 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:46.651492 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:47.489791 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:47.489738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:48.240467 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:48.240428 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:49.987817 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:49.987775 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:50.379373 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:50.379280 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:50.549297 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:50.549264 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:51.050802 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:51.050766 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:51.211559 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:51.211513 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:51.622700 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:51.622669 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:52.865712 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:52.865666 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc"] Nov 02 14:59:52.866243 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:52.866086 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" podUID="d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" containerName="maas-api" containerID="cri-o://6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b" gracePeriod=30 Nov 02 14:59:52.982151 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:52.982119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:53.127530 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.127447 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:53.182412 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.182380 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-kube-api-access-mk9l6\" (UniqueName: \"kubernetes.io/projected/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-kube-api-access-mk9l6\") pod \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " Nov 02 14:59:53.182625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.182437 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-metrics-certs\") pod \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " Nov 02 14:59:53.182625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.182479 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-openshift-service-ca\") pod \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " Nov 02 14:59:53.182625 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.182504 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-maas-api-tls\") pod \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\" (UID: \"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd\") " Nov 02 14:59:53.182936 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.182903 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-openshift-service-ca" pod "d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" (UID: "d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:59:53.184577 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.184553 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-maas-api-tls" pod "d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" (UID: "d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:59:53.185108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.185079 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-metrics-certs" pod "d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" (UID: "d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:59:53.185108 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.185092 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-kube-api-access-mk9l6" pod "d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" (UID: "d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd"). InnerVolumeSpecName "kube-api-access-mk9l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:59:53.286045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.285946 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk9l6\" (UniqueName: \"kubernetes.io/projected/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-kube-api-access-mk9l6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:53.286045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.285999 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:53.286045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.286015 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:53.286045 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.286029 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 14:59:53.324611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.324358 5591 generic.go:356] "Generic (PLEG): container finished" podID="d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" containerID="6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b" exitCode=0 Nov 02 14:59:53.324611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.324409 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" Nov 02 14:59:53.324611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.324410 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" event={"ID":"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd","Type":"ContainerDied","Data":"6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b"} Nov 02 14:59:53.324611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.324465 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc" event={"ID":"d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd","Type":"ContainerDied","Data":"40a66f35711070ef31c4733e4a5e8b7cae88c228d2694e28987f49dbb295b44d"} Nov 02 14:59:53.324611 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.324495 5591 scope.go:122] "RemoveContainer" containerID="6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b" Nov 02 14:59:53.339334 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.339300 5591 scope.go:122] "RemoveContainer" containerID="6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b" Nov 02 14:59:53.339696 ip-10-0-140-236 kubenswrapper[5591]: E1102 14:59:53.339675 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b\": container with ID starting with 6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b not found: ID does not exist" containerID="6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b" Nov 02 14:59:53.339768 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.339704 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b"} err="failed to get container status \"6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b\": rpc error: code = NotFound desc = could not find container \"6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b\": container with ID starting with 6918c909c081678b68f39aadc14767d288c3c8c27ec878a3e7982cdd320db79b not found: ID does not exist" Nov 02 14:59:53.374634 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.374603 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc"] Nov 02 14:59:53.378837 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.378775 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fcad4d9a-68d59999c7-wd8xc"] Nov 02 14:59:53.428074 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.428037 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" path="/var/lib/kubelet/pods/d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd/volumes" Nov 02 14:59:53.521549 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.521512 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:53.681516 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:53.681481 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:55.008555 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:55.008518 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 14:59:56.833104 ip-10-0-140-236 kubenswrapper[5591]: I1102 14:59:56.833063 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:01.073133 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:00:01.073093 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:00:02.377686 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.377646 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-7c5m7"] Nov 02 15:00:02.378117 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.378055 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" containerName="maas-api" Nov 02 15:00:02.378117 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.378068 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" containerName="maas-api" Nov 02 15:00:02.378200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.378142 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c415b9-27f5-4f3c-85d5-a7cba8c71ccd" containerName="maas-api" Nov 02 15:00:02.382625 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.382590 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.401666 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.401627 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 15:00:02.412539 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.412496 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-7c5m7"] Nov 02 15:00:02.413867 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.413842 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jtrps\"" Nov 02 15:00:02.413867 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.413856 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 15:00:02.414099 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.413890 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 15:00:02.476771 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.476727 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tg7\" (UniqueName: \"kubernetes.io/projected/4e8c3599-83c2-4250-bdfe-8e184231591f-kube-api-access-j4tg7\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.476975 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.476784 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-webhook-cert\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.477078 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.477008 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-metrics-certs\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.577793 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.577754 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tg7\" (UniqueName: \"kubernetes.io/projected/4e8c3599-83c2-4250-bdfe-8e184231591f-kube-api-access-j4tg7\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.577793 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.577797 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-webhook-cert\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.578109 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.577875 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-metrics-certs\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.580504 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.580474 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-metrics-certs\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.580661 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.580548 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-webhook-cert\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.601146 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.601110 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tg7\" (UniqueName: \"kubernetes.io/projected/4e8c3599-83c2-4250-bdfe-8e184231591f-kube-api-access-j4tg7\") pod \"maas-controller-7f746cf4d4-7c5m7\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:02.698975 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:02.698931 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:03.049051 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:03.041695 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-7c5m7"] Nov 02 15:00:03.064863 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:00:03.064782 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8c3599_83c2_4250_bdfe_8e184231591f.slice/crio-559dda9202b22d7e4caa8dd73bc5904f80632f46a716fd3012c45d6a057b7e56 WatchSource:0}: Error finding container 559dda9202b22d7e4caa8dd73bc5904f80632f46a716fd3012c45d6a057b7e56: Status 404 returned error can't find the container with id 559dda9202b22d7e4caa8dd73bc5904f80632f46a716fd3012c45d6a057b7e56 Nov 02 15:00:03.375397 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:03.375289 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" event={"ID":"4e8c3599-83c2-4250-bdfe-8e184231591f","Type":"ContainerStarted","Data":"559dda9202b22d7e4caa8dd73bc5904f80632f46a716fd3012c45d6a057b7e56"} Nov 02 15:00:04.381723 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:04.381685 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" event={"ID":"4e8c3599-83c2-4250-bdfe-8e184231591f","Type":"ContainerStarted","Data":"60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377"} Nov 02 15:00:04.382185 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:04.381787 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:04.425534 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:04.425469 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" podStartSLOduration=1.614053378 podStartE2EDuration="2.425450908s" totalImagesPullingTime="811.39753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:04.424322741 +0000 UTC m=+4553.149944511" watchObservedRunningTime="2026-11-02 15:00:04.425450908 +0000 UTC m=+4553.151072680" Nov 02 15:00:15.388792 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:15.388761 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:27.557442 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.557400 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh"] Nov 02 15:00:27.561300 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.561273 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.590736 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.590703 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh"] Nov 02 15:00:27.722128 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722093 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.722128 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722131 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.722351 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722164 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.722351 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722200 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.722351 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722279 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.722351 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722299 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.722351 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722340 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.722546 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722445 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52sbv\" (UniqueName: \"kubernetes.io/projected/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-kube-api-access-52sbv\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.722546 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.722499 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.823977 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.823886 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52sbv\" (UniqueName: \"kubernetes.io/projected/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-kube-api-access-52sbv\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.823977 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.823944 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824012 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824041 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824068 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824142 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824207 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824239 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824622 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824309 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824622 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824527 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824881 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824858 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824967 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824912 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.824967 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.824930 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.825445 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.825422 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.826322 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.826299 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.826668 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.826649 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.846979 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.846947 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.847205 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.847187 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52sbv\" (UniqueName: \"kubernetes.io/projected/ad6d5b6e-e351-4020-81b4-98bb9d0a013a-kube-api-access-52sbv\") pod \"maas-default-gateway-openshift-default-6886b96f9-qqfsh\" (UID: \"ad6d5b6e-e351-4020-81b4-98bb9d0a013a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:27.875150 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:27.875124 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:28.188849 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:28.188810 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh"] Nov 02 15:00:28.192940 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:00:28.192912 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6d5b6e_e351_4020_81b4_98bb9d0a013a.slice/crio-63c8db68b6abdf407f956a347bf33471b59314195888e4fa70533e42b73183ca WatchSource:0}: Error finding container 63c8db68b6abdf407f956a347bf33471b59314195888e4fa70533e42b73183ca: Status 404 returned error can't find the container with id 63c8db68b6abdf407f956a347bf33471b59314195888e4fa70533e42b73183ca Nov 02 15:00:28.195350 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:28.195308 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:00:28.195459 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:28.195392 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:00:28.195459 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:28.195424 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:00:28.497706 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:28.497607 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" event={"ID":"ad6d5b6e-e351-4020-81b4-98bb9d0a013a","Type":"ContainerStarted","Data":"7c23e71793afa14f935103fa6bc126aee90cf3f5bd4b3a49edba193a1da67903"} Nov 02 15:00:28.497706 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:28.497678 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" event={"ID":"ad6d5b6e-e351-4020-81b4-98bb9d0a013a","Type":"ContainerStarted","Data":"63c8db68b6abdf407f956a347bf33471b59314195888e4fa70533e42b73183ca"} Nov 02 15:00:28.546819 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:28.546761 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podStartSLOduration=1.546747328 podStartE2EDuration="1.546747328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:28.544514456 +0000 UTC m=+4577.270136227" watchObservedRunningTime="2026-11-02 15:00:28.546747328 +0000 UTC m=+4577.272369094" Nov 02 15:00:28.875316 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:28.875201 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:29.310429 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:29.310388 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:29.877733 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:29.877696 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 15:00:29.878147 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:29.877750 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 15:00:30.875749 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:30.875711 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 15:00:30.875936 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:30.875783 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 15:00:31.080975 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:00:31.080937 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:00:31.876594 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:31.876554 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 15:00:31.876795 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:31.876611 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 15:00:32.876314 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:32.876271 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 15:00:32.876896 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:32.876345 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 15:00:33.458634 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.458590 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:33.527359 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.527324 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-7c5m7"] Nov 02 15:00:33.527588 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.527549 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" podUID="4e8c3599-83c2-4250-bdfe-8e184231591f" containerName="manager" containerID="cri-o://60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377" gracePeriod=30 Nov 02 15:00:33.810134 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.810105 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:33.875473 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.875438 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 15:00:33.875657 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.875495 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 15:00:33.991339 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.991256 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-webhook-cert\") pod \"4e8c3599-83c2-4250-bdfe-8e184231591f\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " Nov 02 15:00:33.991339 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.991303 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-metrics-certs\") pod \"4e8c3599-83c2-4250-bdfe-8e184231591f\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " Nov 02 15:00:33.991798 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.991455 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8c3599-83c2-4250-bdfe-8e184231591f-kube-api-access-j4tg7\" (UniqueName: \"kubernetes.io/projected/4e8c3599-83c2-4250-bdfe-8e184231591f-kube-api-access-j4tg7\") pod \"4e8c3599-83c2-4250-bdfe-8e184231591f\" (UID: \"4e8c3599-83c2-4250-bdfe-8e184231591f\") " Nov 02 15:00:33.993485 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.993459 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-webhook-cert" pod "4e8c3599-83c2-4250-bdfe-8e184231591f" (UID: "4e8c3599-83c2-4250-bdfe-8e184231591f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:00:33.993485 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.993477 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-metrics-certs" pod "4e8c3599-83c2-4250-bdfe-8e184231591f" (UID: "4e8c3599-83c2-4250-bdfe-8e184231591f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:00:33.993658 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:33.993574 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8c3599-83c2-4250-bdfe-8e184231591f-kube-api-access-j4tg7" pod "4e8c3599-83c2-4250-bdfe-8e184231591f" (UID: "4e8c3599-83c2-4250-bdfe-8e184231591f"). InnerVolumeSpecName "kube-api-access-j4tg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:00:34.092682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.092649 5591 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-webhook-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:00:34.092682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.092678 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8c3599-83c2-4250-bdfe-8e184231591f-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:00:34.092682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.092687 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tg7\" (UniqueName: \"kubernetes.io/projected/4e8c3599-83c2-4250-bdfe-8e184231591f-kube-api-access-j4tg7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:00:34.532169 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.532128 5591 generic.go:356] "Generic (PLEG): container finished" podID="4e8c3599-83c2-4250-bdfe-8e184231591f" containerID="60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377" exitCode=0 Nov 02 15:00:34.532380 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.532190 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" Nov 02 15:00:34.532380 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.532211 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" event={"ID":"4e8c3599-83c2-4250-bdfe-8e184231591f","Type":"ContainerDied","Data":"60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377"} Nov 02 15:00:34.532380 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.532300 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-7c5m7" event={"ID":"4e8c3599-83c2-4250-bdfe-8e184231591f","Type":"ContainerDied","Data":"559dda9202b22d7e4caa8dd73bc5904f80632f46a716fd3012c45d6a057b7e56"} Nov 02 15:00:34.532380 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.532329 5591 scope.go:122] "RemoveContainer" containerID="60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377" Nov 02 15:00:34.541893 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.541865 5591 scope.go:122] "RemoveContainer" containerID="60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377" Nov 02 15:00:34.542207 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:00:34.542186 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377\": container with ID starting with 60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377 not found: ID does not exist" containerID="60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377" Nov 02 15:00:34.542288 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.542219 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377"} err="failed to get container status \"60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377\": rpc error: code = NotFound desc = could not find container \"60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377\": container with ID starting with 60babf1d4ba9e0ffcc3b9ad22d767adf6eb9042da58cf141f838e61e11995377 not found: ID does not exist" Nov 02 15:00:34.582274 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.582239 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-7c5m7"] Nov 02 15:00:34.594347 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.594314 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-7c5m7"] Nov 02 15:00:34.875819 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.875730 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 02 15:00:34.875819 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:34.875795 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 02 15:00:35.429446 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.429411 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8c3599-83c2-4250-bdfe-8e184231591f" path="/var/lib/kubelet/pods/4e8c3599-83c2-4250-bdfe-8e184231591f/volumes" Nov 02 15:00:35.769805 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.769700 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:35.856237 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.856200 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kpf4p"] Nov 02 15:00:35.856826 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.856804 5591 cpu_manager.go:414] "containerMap: removing container" podUID="4e8c3599-83c2-4250-bdfe-8e184231591f" containerName="manager" Nov 02 15:00:35.856877 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.856832 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8c3599-83c2-4250-bdfe-8e184231591f" containerName="manager" Nov 02 15:00:35.856969 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.856956 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8c3599-83c2-4250-bdfe-8e184231591f" containerName="manager" Nov 02 15:00:35.861882 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.861859 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:35.890318 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.890274 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jtrps\"" Nov 02 15:00:35.890318 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.890291 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 15:00:35.890571 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.890319 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 15:00:35.890751 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.890710 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 15:00:35.892555 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.892525 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kpf4p"] Nov 02 15:00:35.905906 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.905865 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90e3c4f6-049a-49fe-88e5-b0172072ea87-webhook-cert\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:35.906063 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.905951 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e3c4f6-049a-49fe-88e5-b0172072ea87-metrics-certs\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:35.906164 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:35.906065 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvzk\" (UniqueName: \"kubernetes.io/projected/90e3c4f6-049a-49fe-88e5-b0172072ea87-kube-api-access-pzvzk\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:36.007044 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.006971 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90e3c4f6-049a-49fe-88e5-b0172072ea87-webhook-cert\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:36.007271 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.007069 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e3c4f6-049a-49fe-88e5-b0172072ea87-metrics-certs\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:36.007271 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.007110 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvzk\" (UniqueName: \"kubernetes.io/projected/90e3c4f6-049a-49fe-88e5-b0172072ea87-kube-api-access-pzvzk\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:36.009524 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.009495 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e3c4f6-049a-49fe-88e5-b0172072ea87-metrics-certs\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:36.009687 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.009649 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90e3c4f6-049a-49fe-88e5-b0172072ea87-webhook-cert\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:36.035938 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.035856 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvzk\" (UniqueName: \"kubernetes.io/projected/90e3c4f6-049a-49fe-88e5-b0172072ea87-kube-api-access-pzvzk\") pod \"maas-controller-7f746cf4d4-kpf4p\" (UID: \"90e3c4f6-049a-49fe-88e5-b0172072ea87\") " pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:36.175250 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.175199 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:36.487761 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.487725 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kpf4p"] Nov 02 15:00:36.489066 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.489041 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:36.489537 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.489369 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:36.490354 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.490331 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" Nov 02 15:00:36.491735 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:00:36.491707 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e3c4f6_049a_49fe_88e5_b0172072ea87.slice/crio-ad016539dd31d2640bdf57ff8974fb1b50296ff21abf73b59496e26d2d819a4c WatchSource:0}: Error finding container ad016539dd31d2640bdf57ff8974fb1b50296ff21abf73b59496e26d2d819a4c: Status 404 returned error can't find the container with id ad016539dd31d2640bdf57ff8974fb1b50296ff21abf73b59496e26d2d819a4c Nov 02 15:00:36.543643 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:36.543610 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" event={"ID":"90e3c4f6-049a-49fe-88e5-b0172072ea87","Type":"ContainerStarted","Data":"ad016539dd31d2640bdf57ff8974fb1b50296ff21abf73b59496e26d2d819a4c"} Nov 02 15:00:37.549466 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:37.549429 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" event={"ID":"90e3c4f6-049a-49fe-88e5-b0172072ea87","Type":"ContainerStarted","Data":"f6f30e18ab1534f32868bed765ef12b214ead30bf827957c75d0dd2cbcb7e91b"} Nov 02 15:00:37.549899 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:37.549506 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:37.593370 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:37.593317 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" podStartSLOduration=2.278817446 podStartE2EDuration="2.593303494s" totalImagesPullingTime="314.486048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:37.592000618 +0000 UTC m=+4586.317622387" watchObservedRunningTime="2026-11-02 15:00:37.593303494 +0000 UTC m=+4586.318925263" Nov 02 15:00:48.557167 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:48.557129 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f746cf4d4-kpf4p" Nov 02 15:00:56.100068 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:00:56.099963 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:01:00.778051 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.778015 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c"] Nov 02 15:01:00.781906 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.781885 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.801292 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.801251 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c"] Nov 02 15:01:00.811681 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.811652 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:01:00.811837 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.811685 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:01:00.811837 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.811689 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:01:00.811837 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.811750 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:01:00.840691 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.840664 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.840843 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.840728 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.840900 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.840842 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbmt\" (UniqueName: \"kubernetes.io/projected/a0d4b372-786a-4683-ac87-bd97c991d100-kube-api-access-8bbmt\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.942184 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.942144 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.942402 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.942223 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.942402 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.942257 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbmt\" (UniqueName: \"kubernetes.io/projected/a0d4b372-786a-4683-ac87-bd97c991d100-kube-api-access-8bbmt\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.942905 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.942876 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.944611 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.944588 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:00.965614 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:00.965584 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbmt\" (UniqueName: \"kubernetes.io/projected/a0d4b372-786a-4683-ac87-bd97c991d100-kube-api-access-8bbmt\") pod \"maas-api-revoke-sub-f904bff9-65h5c\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:01.073594 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:01:01.073516 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:01:01.094836 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:01.094805 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:01.391126 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:01.391090 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c"] Nov 02 15:01:01.394793 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:01:01.394760 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d4b372_786a_4683_ac87_bd97c991d100.slice/crio-eec06d0071502bf0dd71d4b85917ed8a5551964e0c053a5a115dfe500230c581 WatchSource:0}: Error finding container eec06d0071502bf0dd71d4b85917ed8a5551964e0c053a5a115dfe500230c581: Status 404 returned error can't find the container with id eec06d0071502bf0dd71d4b85917ed8a5551964e0c053a5a115dfe500230c581 Nov 02 15:01:01.701006 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:01.700950 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" event={"ID":"a0d4b372-786a-4683-ac87-bd97c991d100","Type":"ContainerStarted","Data":"06f9e2eba48c0160eaf50276d41cbc0ba2eccc5822d90b0748d6995d79eb7dcf"} Nov 02 15:01:01.701200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:01.701033 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" event={"ID":"a0d4b372-786a-4683-ac87-bd97c991d100","Type":"ContainerStarted","Data":"eec06d0071502bf0dd71d4b85917ed8a5551964e0c053a5a115dfe500230c581"} Nov 02 15:01:01.752995 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:01.752936 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" podStartSLOduration=1.7529219120000001 podStartE2EDuration="1.752921912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:01.740728673 +0000 UTC m=+4610.466350443" watchObservedRunningTime="2026-11-02 15:01:01.752921912 +0000 UTC m=+4610.478543676" Nov 02 15:01:02.707922 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:02.707885 5591 generic.go:356] "Generic (PLEG): container finished" podID="a0d4b372-786a-4683-ac87-bd97c991d100" containerID="06f9e2eba48c0160eaf50276d41cbc0ba2eccc5822d90b0748d6995d79eb7dcf" exitCode=0 Nov 02 15:01:02.708411 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:02.707972 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" event={"ID":"a0d4b372-786a-4683-ac87-bd97c991d100","Type":"ContainerDied","Data":"06f9e2eba48c0160eaf50276d41cbc0ba2eccc5822d90b0748d6995d79eb7dcf"} Nov 02 15:01:03.844954 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:03.844923 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:03.970027 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:03.969909 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-xks-ca\") pod \"a0d4b372-786a-4683-ac87-bd97c991d100\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " Nov 02 15:01:03.970193 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:03.970030 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-openshift-ca\") pod \"a0d4b372-786a-4683-ac87-bd97c991d100\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " Nov 02 15:01:03.970193 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:03.970068 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d4b372-786a-4683-ac87-bd97c991d100-kube-api-access-8bbmt\" (UniqueName: \"kubernetes.io/projected/a0d4b372-786a-4683-ac87-bd97c991d100-kube-api-access-8bbmt\") pod \"a0d4b372-786a-4683-ac87-bd97c991d100\" (UID: \"a0d4b372-786a-4683-ac87-bd97c991d100\") " Nov 02 15:01:03.970394 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:03.970369 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-openshift-ca" pod "a0d4b372-786a-4683-ac87-bd97c991d100" (UID: "a0d4b372-786a-4683-ac87-bd97c991d100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:01:03.972166 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:03.972139 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-xks-ca" pod "a0d4b372-786a-4683-ac87-bd97c991d100" (UID: "a0d4b372-786a-4683-ac87-bd97c991d100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:01:03.972287 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:03.972270 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d4b372-786a-4683-ac87-bd97c991d100-kube-api-access-8bbmt" pod "a0d4b372-786a-4683-ac87-bd97c991d100" (UID: "a0d4b372-786a-4683-ac87-bd97c991d100"). InnerVolumeSpecName "kube-api-access-8bbmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:01:04.071631 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:04.071585 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:04.071631 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:04.071626 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4b372-786a-4683-ac87-bd97c991d100-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:04.071631 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:04.071639 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbmt\" (UniqueName: \"kubernetes.io/projected/a0d4b372-786a-4683-ac87-bd97c991d100-kube-api-access-8bbmt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:04.717235 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:04.717204 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" Nov 02 15:01:04.717417 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:04.717246 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c" event={"ID":"a0d4b372-786a-4683-ac87-bd97c991d100","Type":"ContainerDied","Data":"eec06d0071502bf0dd71d4b85917ed8a5551964e0c053a5a115dfe500230c581"} Nov 02 15:01:04.717417 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:04.717292 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eec06d0071502bf0dd71d4b85917ed8a5551964e0c053a5a115dfe500230c581" Nov 02 15:01:09.465145 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:09.465111 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:09.627192 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:09.627158 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:12.037303 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:12.037265 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:01:19.318438 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:19.318397 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:01:24.525443 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:24.525401 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:24.701858 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:24.701640 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:25.049030 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:25.048977 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:01:26.405187 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.405151 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9"] Nov 02 15:01:26.405593 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.405561 5591 cpu_manager.go:414] "containerMap: removing container" podUID="a0d4b372-786a-4683-ac87-bd97c991d100" containerName="revoke-keys" Nov 02 15:01:26.405593 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.405575 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d4b372-786a-4683-ac87-bd97c991d100" containerName="revoke-keys" Nov 02 15:01:26.405672 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.405662 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d4b372-786a-4683-ac87-bd97c991d100" containerName="revoke-keys" Nov 02 15:01:26.408855 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.408839 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.426847 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.426817 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:01:26.429494 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.429463 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9"] Nov 02 15:01:26.429797 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.429771 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.429923 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.429852 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6np4\" (UniqueName: \"kubernetes.io/projected/925c0f4d-7bb4-4580-8c85-f3b899735d7f-kube-api-access-b6np4\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.430023 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.429962 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.439026 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.438970 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:01:26.439219 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.439134 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:01:26.439478 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.439452 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:01:26.530949 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.530912 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.531181 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.530966 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6np4\" (UniqueName: \"kubernetes.io/projected/925c0f4d-7bb4-4580-8c85-f3b899735d7f-kube-api-access-b6np4\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.531181 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.531051 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.531699 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.531674 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.533272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.533245 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.554444 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.554416 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6np4\" (UniqueName: \"kubernetes.io/projected/925c0f4d-7bb4-4580-8c85-f3b899735d7f-kube-api-access-b6np4\") pod \"maas-api-revoke-sub-f03629db-pn2m9\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:26.734967 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:26.734885 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:27.033408 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:27.033378 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9"] Nov 02 15:01:27.036197 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:01:27.036162 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925c0f4d_7bb4_4580_8c85_f3b899735d7f.slice/crio-c0262042ff4cfc77d989a14de1dd86d37069c8598ffdbff970e20c9a77d8d3db WatchSource:0}: Error finding container c0262042ff4cfc77d989a14de1dd86d37069c8598ffdbff970e20c9a77d8d3db: Status 404 returned error can't find the container with id c0262042ff4cfc77d989a14de1dd86d37069c8598ffdbff970e20c9a77d8d3db Nov 02 15:01:27.884000 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:27.883956 5591 generic.go:356] "Generic (PLEG): container finished" podID="925c0f4d-7bb4-4580-8c85-f3b899735d7f" containerID="57d5ad75f8aa56ded02d9860842ae685c80c8536d2833f8a215a41ca5b9c9319" exitCode=0 Nov 02 15:01:27.884395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:27.884001 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" event={"ID":"925c0f4d-7bb4-4580-8c85-f3b899735d7f","Type":"ContainerDied","Data":"57d5ad75f8aa56ded02d9860842ae685c80c8536d2833f8a215a41ca5b9c9319"} Nov 02 15:01:27.884395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:27.884055 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" event={"ID":"925c0f4d-7bb4-4580-8c85-f3b899735d7f","Type":"ContainerStarted","Data":"c0262042ff4cfc77d989a14de1dd86d37069c8598ffdbff970e20c9a77d8d3db"} Nov 02 15:01:29.019664 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.019636 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:29.056799 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.056758 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-openshift-ca\") pod \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " Nov 02 15:01:29.057026 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.056894 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-xks-ca\") pod \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " Nov 02 15:01:29.057026 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.056955 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925c0f4d-7bb4-4580-8c85-f3b899735d7f-kube-api-access-b6np4\" (UniqueName: \"kubernetes.io/projected/925c0f4d-7bb4-4580-8c85-f3b899735d7f-kube-api-access-b6np4\") pod \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\" (UID: \"925c0f4d-7bb4-4580-8c85-f3b899735d7f\") " Nov 02 15:01:29.057193 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.057168 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-openshift-ca" pod "925c0f4d-7bb4-4580-8c85-f3b899735d7f" (UID: "925c0f4d-7bb4-4580-8c85-f3b899735d7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:01:29.057318 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.057302 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:29.059257 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.059223 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-xks-ca" pod "925c0f4d-7bb4-4580-8c85-f3b899735d7f" (UID: "925c0f4d-7bb4-4580-8c85-f3b899735d7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:01:29.059384 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.059338 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925c0f4d-7bb4-4580-8c85-f3b899735d7f-kube-api-access-b6np4" pod "925c0f4d-7bb4-4580-8c85-f3b899735d7f" (UID: "925c0f4d-7bb4-4580-8c85-f3b899735d7f"). InnerVolumeSpecName "kube-api-access-b6np4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:01:29.158852 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.158754 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/925c0f4d-7bb4-4580-8c85-f3b899735d7f-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:29.158852 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.158793 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6np4\" (UniqueName: \"kubernetes.io/projected/925c0f4d-7bb4-4580-8c85-f3b899735d7f-kube-api-access-b6np4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:29.318391 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.318351 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:01:29.418283 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.418182 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:29.895668 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.895635 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" Nov 02 15:01:29.895668 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.895648 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9" event={"ID":"925c0f4d-7bb4-4580-8c85-f3b899735d7f","Type":"ContainerDied","Data":"c0262042ff4cfc77d989a14de1dd86d37069c8598ffdbff970e20c9a77d8d3db"} Nov 02 15:01:29.895897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:29.895692 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0262042ff4cfc77d989a14de1dd86d37069c8598ffdbff970e20c9a77d8d3db" Nov 02 15:01:31.073377 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:01:31.073338 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:01:34.128548 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:34.128512 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:42.810998 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.810951 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt"] Nov 02 15:01:42.811517 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.811350 5591 cpu_manager.go:414] "containerMap: removing container" podUID="925c0f4d-7bb4-4580-8c85-f3b899735d7f" containerName="revoke-keys" Nov 02 15:01:42.811517 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.811362 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925c0f4d-7bb4-4580-8c85-f3b899735d7f" containerName="revoke-keys" Nov 02 15:01:42.811517 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.811451 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="925c0f4d-7bb4-4580-8c85-f3b899735d7f" containerName="revoke-keys" Nov 02 15:01:42.815632 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.815613 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:42.832855 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.832827 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:01:42.835249 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.835226 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt"] Nov 02 15:01:42.843913 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.843886 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:01:42.843913 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.843886 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:01:42.844105 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.843924 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:01:42.903202 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.903164 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mrz9\" (UniqueName: \"kubernetes.io/projected/ce753aa7-2155-4bfc-ba8e-b5629124d180-kube-api-access-4mrz9\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:42.903415 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.903231 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:42.903490 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:42.903424 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:43.004574 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.004533 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:43.004784 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.004594 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mrz9\" (UniqueName: \"kubernetes.io/projected/ce753aa7-2155-4bfc-ba8e-b5629124d180-kube-api-access-4mrz9\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:43.004784 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.004647 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:43.005419 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.005386 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:43.006934 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.006911 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:43.027679 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.027650 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mrz9\" (UniqueName: \"kubernetes.io/projected/ce753aa7-2155-4bfc-ba8e-b5629124d180-kube-api-access-4mrz9\") pod \"maas-api-revoke-sub-0165d7d2-6hktt\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:43.128895 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.128810 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:43.429454 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.429426 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt"] Nov 02 15:01:43.430636 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:01:43.430607 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce753aa7_2155_4bfc_ba8e_b5629124d180.slice/crio-e06fa44a3750093ca15db8cfe477dac73987fca9675fb4532c04d0529aeecbbb WatchSource:0}: Error finding container e06fa44a3750093ca15db8cfe477dac73987fca9675fb4532c04d0529aeecbbb: Status 404 returned error can't find the container with id e06fa44a3750093ca15db8cfe477dac73987fca9675fb4532c04d0529aeecbbb Nov 02 15:01:43.968559 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.968514 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" event={"ID":"ce753aa7-2155-4bfc-ba8e-b5629124d180","Type":"ContainerStarted","Data":"7914eb45270e81e15ea4f8b8f013293fb83a895a3fbfb2c3f330f66ed120bd2e"} Nov 02 15:01:43.968961 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:43.968607 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" event={"ID":"ce753aa7-2155-4bfc-ba8e-b5629124d180","Type":"ContainerStarted","Data":"e06fa44a3750093ca15db8cfe477dac73987fca9675fb4532c04d0529aeecbbb"} Nov 02 15:01:44.009588 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:44.009495 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" podStartSLOduration=2.009482337 podStartE2EDuration="2.009482337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:44.008026936 +0000 UTC m=+4652.733648703" watchObservedRunningTime="2026-11-02 15:01:44.009482337 +0000 UTC m=+4652.735104106" Nov 02 15:01:44.978140 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:44.978096 5591 generic.go:356] "Generic (PLEG): container finished" podID="ce753aa7-2155-4bfc-ba8e-b5629124d180" containerID="7914eb45270e81e15ea4f8b8f013293fb83a895a3fbfb2c3f330f66ed120bd2e" exitCode=0 Nov 02 15:01:44.978571 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:44.978186 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" event={"ID":"ce753aa7-2155-4bfc-ba8e-b5629124d180","Type":"ContainerDied","Data":"7914eb45270e81e15ea4f8b8f013293fb83a895a3fbfb2c3f330f66ed120bd2e"} Nov 02 15:01:46.111257 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.111230 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:46.237485 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.237407 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-xks-ca\") pod \"ce753aa7-2155-4bfc-ba8e-b5629124d180\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " Nov 02 15:01:46.237485 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.237466 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce753aa7-2155-4bfc-ba8e-b5629124d180-kube-api-access-4mrz9\" (UniqueName: \"kubernetes.io/projected/ce753aa7-2155-4bfc-ba8e-b5629124d180-kube-api-access-4mrz9\") pod \"ce753aa7-2155-4bfc-ba8e-b5629124d180\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " Nov 02 15:01:46.237777 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.237604 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-openshift-ca\") pod \"ce753aa7-2155-4bfc-ba8e-b5629124d180\" (UID: \"ce753aa7-2155-4bfc-ba8e-b5629124d180\") " Nov 02 15:01:46.237960 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.237935 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-openshift-ca" pod "ce753aa7-2155-4bfc-ba8e-b5629124d180" (UID: "ce753aa7-2155-4bfc-ba8e-b5629124d180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:01:46.239518 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.239495 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-xks-ca" pod "ce753aa7-2155-4bfc-ba8e-b5629124d180" (UID: "ce753aa7-2155-4bfc-ba8e-b5629124d180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:01:46.239601 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.239571 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce753aa7-2155-4bfc-ba8e-b5629124d180-kube-api-access-4mrz9" pod "ce753aa7-2155-4bfc-ba8e-b5629124d180" (UID: "ce753aa7-2155-4bfc-ba8e-b5629124d180"). InnerVolumeSpecName "kube-api-access-4mrz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:01:46.339283 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.339240 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:46.339283 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.339277 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mrz9\" (UniqueName: \"kubernetes.io/projected/ce753aa7-2155-4bfc-ba8e-b5629124d180-kube-api-access-4mrz9\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:46.339283 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.339291 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce753aa7-2155-4bfc-ba8e-b5629124d180-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:46.987890 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.987859 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" Nov 02 15:01:46.988100 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.987886 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt" event={"ID":"ce753aa7-2155-4bfc-ba8e-b5629124d180","Type":"ContainerDied","Data":"e06fa44a3750093ca15db8cfe477dac73987fca9675fb4532c04d0529aeecbbb"} Nov 02 15:01:46.988100 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:46.987931 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e06fa44a3750093ca15db8cfe477dac73987fca9675fb4532c04d0529aeecbbb" Nov 02 15:01:53.710532 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.710487 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh"] Nov 02 15:01:53.711096 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.711075 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ce753aa7-2155-4bfc-ba8e-b5629124d180" containerName="revoke-keys" Nov 02 15:01:53.711195 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.711100 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce753aa7-2155-4bfc-ba8e-b5629124d180" containerName="revoke-keys" Nov 02 15:01:53.711257 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.711234 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce753aa7-2155-4bfc-ba8e-b5629124d180" containerName="revoke-keys" Nov 02 15:01:53.714184 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.714161 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.731190 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.731160 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:01:53.731435 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.731302 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:01:53.743256 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.743226 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:01:53.743408 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.743296 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:01:53.745191 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.745159 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh"] Nov 02 15:01:53.807580 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.807537 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.807764 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.807594 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j476m\" (UniqueName: \"kubernetes.io/projected/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-kube-api-access-j476m\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.807764 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.807677 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.908700 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.908662 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.908864 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.908711 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j476m\" (UniqueName: \"kubernetes.io/projected/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-kube-api-access-j476m\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.908935 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.908878 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.909405 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.909382 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.911223 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.911205 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:53.932227 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:53.932197 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j476m\" (UniqueName: \"kubernetes.io/projected/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-kube-api-access-j476m\") pod \"maas-api-revoke-sub-6b19338f-7zvvh\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:54.027806 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:54.027727 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:54.324084 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:54.324060 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh"] Nov 02 15:01:54.326999 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:01:54.326953 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0df51f_70f8_4ca4_ab65_a0dc0201bb7c.slice/crio-5185d661b88969d9bfd396f0d5ad66bc8a4a3a521f10320e5038b2a868799ba7 WatchSource:0}: Error finding container 5185d661b88969d9bfd396f0d5ad66bc8a4a3a521f10320e5038b2a868799ba7: Status 404 returned error can't find the container with id 5185d661b88969d9bfd396f0d5ad66bc8a4a3a521f10320e5038b2a868799ba7 Nov 02 15:01:55.025972 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:55.025870 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" event={"ID":"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c","Type":"ContainerStarted","Data":"d697cd379063f41824145bf21d03e1f9202499585d25b350dacbc15ea54d3c4d"} Nov 02 15:01:55.025972 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:55.025951 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" event={"ID":"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c","Type":"ContainerStarted","Data":"5185d661b88969d9bfd396f0d5ad66bc8a4a3a521f10320e5038b2a868799ba7"} Nov 02 15:01:55.066476 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:55.066403 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" podStartSLOduration=2.066388238 podStartE2EDuration="2.066388238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:55.065105456 +0000 UTC m=+4663.790727226" watchObservedRunningTime="2026-11-02 15:01:55.066388238 +0000 UTC m=+4663.792010004" Nov 02 15:01:56.032150 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:56.032114 5591 generic.go:356] "Generic (PLEG): container finished" podID="2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c" containerID="d697cd379063f41824145bf21d03e1f9202499585d25b350dacbc15ea54d3c4d" exitCode=0 Nov 02 15:01:56.032774 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:56.032180 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" event={"ID":"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c","Type":"ContainerDied","Data":"d697cd379063f41824145bf21d03e1f9202499585d25b350dacbc15ea54d3c4d"} Nov 02 15:01:57.171251 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.171223 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:01:57.238071 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.238037 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-xks-ca\") pod \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " Nov 02 15:01:57.238244 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.238168 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-openshift-ca\") pod \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " Nov 02 15:01:57.238244 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.238221 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-kube-api-access-j476m\" (UniqueName: \"kubernetes.io/projected/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-kube-api-access-j476m\") pod \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\" (UID: \"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c\") " Nov 02 15:01:57.238527 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.238497 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-openshift-ca" pod "2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c" (UID: "2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:01:57.240379 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.240351 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-xks-ca" pod "2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c" (UID: "2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:01:57.240499 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.240389 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-kube-api-access-j476m" pod "2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c" (UID: "2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c"). InnerVolumeSpecName "kube-api-access-j476m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:01:57.339333 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.339243 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j476m\" (UniqueName: \"kubernetes.io/projected/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-kube-api-access-j476m\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:57.339333 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.339276 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:57.339333 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:57.339287 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:01:58.041579 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:58.041542 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" event={"ID":"2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c","Type":"ContainerDied","Data":"5185d661b88969d9bfd396f0d5ad66bc8a4a3a521f10320e5038b2a868799ba7"} Nov 02 15:01:58.041744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:58.041592 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5185d661b88969d9bfd396f0d5ad66bc8a4a3a521f10320e5038b2a868799ba7" Nov 02 15:01:58.041744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:01:58.041559 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh" Nov 02 15:02:01.073197 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:02:01.073161 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:02:05.057121 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.057081 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:02:05.610005 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.609452 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj"] Nov 02 15:02:05.610005 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.609820 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c" containerName="revoke-keys" Nov 02 15:02:05.610005 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.609831 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c" containerName="revoke-keys" Nov 02 15:02:05.610005 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.609914 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c" containerName="revoke-keys" Nov 02 15:02:05.613152 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.613127 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.655009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.649009 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj"] Nov 02 15:02:05.655009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.649225 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4711881d-kserve-self-signed-certs\"" Nov 02 15:02:05.812129 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.812082 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-home\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.812439 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.812409 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-model-cache\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.812587 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.812572 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c46f606-8025-4f73-92f9-132f072fb760-tls-certs\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.812725 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.812711 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-dshm\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.812851 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.812836 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nswq\" (UniqueName: \"kubernetes.io/projected/7c46f606-8025-4f73-92f9-132f072fb760-kube-api-access-6nswq\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.813058 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.813040 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-tmp-dir\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.815163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.815112 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:05.914537 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.914494 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-home\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.914800 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.914783 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-model-cache\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.914914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.914901 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c46f606-8025-4f73-92f9-132f072fb760-tls-certs\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.915032 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.915021 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-dshm\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.915185 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.915174 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nswq\" (UniqueName: \"kubernetes.io/projected/7c46f606-8025-4f73-92f9-132f072fb760-kube-api-access-6nswq\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.915319 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.915304 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-tmp-dir\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.915563 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.915531 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-model-cache\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.916122 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.915561 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-home\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.916234 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.915708 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-tmp-dir\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.918282 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.918256 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-dshm\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.920168 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.920144 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c46f606-8025-4f73-92f9-132f072fb760-tls-certs\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:05.948786 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:05.948745 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nswq\" (UniqueName: \"kubernetes.io/projected/7c46f606-8025-4f73-92f9-132f072fb760-kube-api-access-6nswq\") pod \"e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:06.011243 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:06.011203 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:06.122759 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:06.122695 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:06.496060 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:06.480877 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj"] Nov 02 15:02:07.128857 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:07.128819 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" event={"ID":"7c46f606-8025-4f73-92f9-132f072fb760","Type":"ContainerStarted","Data":"b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43"} Nov 02 15:02:07.128857 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:07.128862 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" event={"ID":"7c46f606-8025-4f73-92f9-132f072fb760","Type":"ContainerStarted","Data":"3804d415c6e1485305bbd8c0c83d275369f137fe4a352d614be9bcc56d92689b"} Nov 02 15:02:08.012199 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:08.012119 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:08.807610 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:08.807568 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:11.133384 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:11.133323 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" podStartSLOduration=6.133307402 podStartE2EDuration="6.133307402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:07.191261553 +0000 UTC m=+4675.916883324" watchObservedRunningTime="2026-11-02 15:02:11.133307402 +0000 UTC m=+4679.858929197" Nov 02 15:02:11.134247 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:11.134227 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:02:16.012591 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:16.012553 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:16.025192 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:16.025164 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:16.216549 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:16.216517 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:02:29.537629 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:29.537531 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:31.073040 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:02:31.072977 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:02:47.085554 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:47.085514 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:49.416301 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:49.416265 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:02:51.100685 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:51.100645 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:52.541132 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:52.541089 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:02:52.541793 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:52.541761 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:02:57.061182 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.061143 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m"] Nov 02 15:02:57.064681 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.064656 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.084806 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.084771 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:02:57.098432 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.098396 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:02:57.098432 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.098418 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:02:57.098642 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.098558 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:02:57.101311 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.101283 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m"] Nov 02 15:02:57.147779 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.147745 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.147978 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.147786 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.147978 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.147867 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2j49\" (UniqueName: \"kubernetes.io/projected/68874cb9-a420-45c2-a1b1-f332d4494a16-kube-api-access-w2j49\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.248728 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.248691 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.248939 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.248737 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.248939 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.248759 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2j49\" (UniqueName: \"kubernetes.io/projected/68874cb9-a420-45c2-a1b1-f332d4494a16-kube-api-access-w2j49\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.249503 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.249474 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.251014 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.250994 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.273874 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.273847 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2j49\" (UniqueName: \"kubernetes.io/projected/68874cb9-a420-45c2-a1b1-f332d4494a16-kube-api-access-w2j49\") pod \"maas-api-revoke-sub-45072ecc-j9s7m\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.377120 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.377040 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:02:57.680922 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:57.680888 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m"] Nov 02 15:02:57.683718 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:02:57.683681 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68874cb9_a420_45c2_a1b1_f332d4494a16.slice/crio-9bf9b282d0a15ec4b8790f6b44e3bff233d0bbb1067069084c59d792705f871c WatchSource:0}: Error finding container 9bf9b282d0a15ec4b8790f6b44e3bff233d0bbb1067069084c59d792705f871c: Status 404 returned error can't find the container with id 9bf9b282d0a15ec4b8790f6b44e3bff233d0bbb1067069084c59d792705f871c Nov 02 15:02:58.421879 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:58.421832 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" event={"ID":"68874cb9-a420-45c2-a1b1-f332d4494a16","Type":"ContainerStarted","Data":"c215c3a38d634d70562dd9bf08bfef63d4456b561eebe9f3fbe6d630fefb8785"} Nov 02 15:02:58.422406 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:58.421916 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" event={"ID":"68874cb9-a420-45c2-a1b1-f332d4494a16","Type":"ContainerStarted","Data":"9bf9b282d0a15ec4b8790f6b44e3bff233d0bbb1067069084c59d792705f871c"} Nov 02 15:02:58.467083 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:58.467032 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" podStartSLOduration=1.467017894 podStartE2EDuration="1.467017894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:58.46549914 +0000 UTC m=+4727.191120910" watchObservedRunningTime="2026-11-02 15:02:58.467017894 +0000 UTC m=+4727.192639663" Nov 02 15:02:59.433392 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:59.433358 5591 generic.go:356] "Generic (PLEG): container finished" podID="68874cb9-a420-45c2-a1b1-f332d4494a16" containerID="c215c3a38d634d70562dd9bf08bfef63d4456b561eebe9f3fbe6d630fefb8785" exitCode=0 Nov 02 15:02:59.434211 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:02:59.434185 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" event={"ID":"68874cb9-a420-45c2-a1b1-f332d4494a16","Type":"ContainerDied","Data":"c215c3a38d634d70562dd9bf08bfef63d4456b561eebe9f3fbe6d630fefb8785"} Nov 02 15:03:00.573350 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.573324 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:03:00.682774 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.682731 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68874cb9-a420-45c2-a1b1-f332d4494a16-kube-api-access-w2j49\" (UniqueName: \"kubernetes.io/projected/68874cb9-a420-45c2-a1b1-f332d4494a16-kube-api-access-w2j49\") pod \"68874cb9-a420-45c2-a1b1-f332d4494a16\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " Nov 02 15:03:00.682960 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.682837 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-xks-ca\") pod \"68874cb9-a420-45c2-a1b1-f332d4494a16\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " Nov 02 15:03:00.682960 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.682894 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-openshift-ca\") pod \"68874cb9-a420-45c2-a1b1-f332d4494a16\" (UID: \"68874cb9-a420-45c2-a1b1-f332d4494a16\") " Nov 02 15:03:00.683319 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.683297 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-openshift-ca" pod "68874cb9-a420-45c2-a1b1-f332d4494a16" (UID: "68874cb9-a420-45c2-a1b1-f332d4494a16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:03:00.685040 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.685002 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68874cb9-a420-45c2-a1b1-f332d4494a16-kube-api-access-w2j49" pod "68874cb9-a420-45c2-a1b1-f332d4494a16" (UID: "68874cb9-a420-45c2-a1b1-f332d4494a16"). InnerVolumeSpecName "kube-api-access-w2j49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:03:00.685154 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.685068 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-xks-ca" pod "68874cb9-a420-45c2-a1b1-f332d4494a16" (UID: "68874cb9-a420-45c2-a1b1-f332d4494a16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:03:00.783794 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.783751 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2j49\" (UniqueName: \"kubernetes.io/projected/68874cb9-a420-45c2-a1b1-f332d4494a16-kube-api-access-w2j49\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:00.783794 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.783787 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:00.783794 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:00.783797 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68874cb9-a420-45c2-a1b1-f332d4494a16-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:01.073642 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:03:01.073548 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:03:01.443781 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:01.443753 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" Nov 02 15:03:01.443927 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:01.443785 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m" event={"ID":"68874cb9-a420-45c2-a1b1-f332d4494a16","Type":"ContainerDied","Data":"9bf9b282d0a15ec4b8790f6b44e3bff233d0bbb1067069084c59d792705f871c"} Nov 02 15:03:01.443927 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:01.443831 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bf9b282d0a15ec4b8790f6b44e3bff233d0bbb1067069084c59d792705f871c" Nov 02 15:03:02.027273 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:02.027231 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh"] Nov 02 15:03:02.040972 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:02.040930 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b5d118-grcqh"] Nov 02 15:03:03.429708 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:03.429668 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78edfcb3-0a81-4845-925c-dc30b7fac4ba" path="/var/lib/kubelet/pods/78edfcb3-0a81-4845-925c-dc30b7fac4ba/volumes" Nov 02 15:03:08.532138 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:08.532101 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:03:08.822614 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:08.822540 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj"] Nov 02 15:03:08.921533 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:08.921490 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:09.083278 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:09.083182 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:11.695287 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.695247 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw"] Nov 02 15:03:11.695765 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.695750 5591 cpu_manager.go:414] "containerMap: removing container" podUID="68874cb9-a420-45c2-a1b1-f332d4494a16" containerName="revoke-keys" Nov 02 15:03:11.695813 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.695767 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68874cb9-a420-45c2-a1b1-f332d4494a16" containerName="revoke-keys" Nov 02 15:03:11.695850 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.695829 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="68874cb9-a420-45c2-a1b1-f332d4494a16" containerName="revoke-keys" Nov 02 15:03:11.700425 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.700403 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.719287 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.719256 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw"] Nov 02 15:03:11.730977 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.730942 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:03:11.730977 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.730962 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:03:11.731187 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.730939 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:03:11.731187 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.731059 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:03:11.795144 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.795107 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8tb\" (UniqueName: \"kubernetes.io/projected/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-kube-api-access-9g8tb\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.795313 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.795157 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.795424 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.795393 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.896917 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.896881 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8tb\" (UniqueName: \"kubernetes.io/projected/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-kube-api-access-9g8tb\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.896917 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.896923 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.897175 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.897008 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.897678 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.897655 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.899290 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.899272 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:11.921821 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:11.921798 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8tb\" (UniqueName: \"kubernetes.io/projected/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-kube-api-access-9g8tb\") pod \"maas-api-revoke-sub-99e058e0-mbntw\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:12.014163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:12.014071 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:12.317496 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:12.317465 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw"] Nov 02 15:03:12.320322 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:03:12.320292 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b74b07e_c864_4cec_85f9_cf1bf0f47cbe.slice/crio-2eaa0324f98740267216b2f75f0886890040b8b0692e1f53be32499e6c1615ee WatchSource:0}: Error finding container 2eaa0324f98740267216b2f75f0886890040b8b0692e1f53be32499e6c1615ee: Status 404 returned error can't find the container with id 2eaa0324f98740267216b2f75f0886890040b8b0692e1f53be32499e6c1615ee Nov 02 15:03:12.500443 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:12.500395 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" event={"ID":"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe","Type":"ContainerStarted","Data":"2eaa0324f98740267216b2f75f0886890040b8b0692e1f53be32499e6c1615ee"} Nov 02 15:03:13.510618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:13.510511 5591 generic.go:356] "Generic (PLEG): container finished" podID="1b74b07e-c864-4cec-85f9-cf1bf0f47cbe" containerID="5a3888785df1fea42b49dd56368588ac6cab075ab900f1cb5bd20eddcbc57633" exitCode=0 Nov 02 15:03:13.510618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:13.510577 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" event={"ID":"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe","Type":"ContainerDied","Data":"5a3888785df1fea42b49dd56368588ac6cab075ab900f1cb5bd20eddcbc57633"} Nov 02 15:03:14.027100 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.027066 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5"] Nov 02 15:03:14.052007 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.044851 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1b6ea-45tg5"] Nov 02 15:03:14.833842 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.833806 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:14.930188 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.930152 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-kube-api-access-9g8tb\" (UniqueName: \"kubernetes.io/projected/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-kube-api-access-9g8tb\") pod \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " Nov 02 15:03:14.930395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.930237 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-openshift-ca\") pod \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " Nov 02 15:03:14.930395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.930282 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-xks-ca\") pod \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\" (UID: \"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe\") " Nov 02 15:03:14.932201 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.932173 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-openshift-ca" pod "1b74b07e-c864-4cec-85f9-cf1bf0f47cbe" (UID: "1b74b07e-c864-4cec-85f9-cf1bf0f47cbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:03:14.956416 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.954150 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-xks-ca" pod "1b74b07e-c864-4cec-85f9-cf1bf0f47cbe" (UID: "1b74b07e-c864-4cec-85f9-cf1bf0f47cbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:03:14.956416 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:14.954710 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-kube-api-access-9g8tb" pod "1b74b07e-c864-4cec-85f9-cf1bf0f47cbe" (UID: "1b74b07e-c864-4cec-85f9-cf1bf0f47cbe"). InnerVolumeSpecName "kube-api-access-9g8tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:03:15.031906 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:15.031870 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:15.031906 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:15.031901 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:15.031906 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:15.031911 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g8tb\" (UniqueName: \"kubernetes.io/projected/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe-kube-api-access-9g8tb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:15.429564 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:15.429526 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68" path="/var/lib/kubelet/pods/e1fc8cc1-c69e-4cb8-9bda-5370b6d65b68/volumes" Nov 02 15:03:15.521194 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:15.521164 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" Nov 02 15:03:15.521376 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:15.521161 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw" event={"ID":"1b74b07e-c864-4cec-85f9-cf1bf0f47cbe","Type":"ContainerDied","Data":"2eaa0324f98740267216b2f75f0886890040b8b0692e1f53be32499e6c1615ee"} Nov 02 15:03:15.521376 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:15.521293 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eaa0324f98740267216b2f75f0886890040b8b0692e1f53be32499e6c1615ee" Nov 02 15:03:16.139475 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:16.139439 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:03:22.543018 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:22.542086 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:03:22.543018 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:22.542156 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:03:23.802460 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:23.802426 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:23.843582 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:23.843548 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" podUID="7c46f606-8025-4f73-92f9-132f072fb760" containerName="main" containerID="cri-o://b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43" gracePeriod=45 Nov 02 15:03:24.096424 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.096396 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:03:24.126072 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.125949 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-tmp-dir\") pod \"7c46f606-8025-4f73-92f9-132f072fb760\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " Nov 02 15:03:24.126072 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126009 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-dshm\") pod \"7c46f606-8025-4f73-92f9-132f072fb760\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " Nov 02 15:03:24.126072 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126038 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-model-cache\") pod \"7c46f606-8025-4f73-92f9-132f072fb760\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " Nov 02 15:03:24.126373 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126096 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-home\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-home\") pod \"7c46f606-8025-4f73-92f9-132f072fb760\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " Nov 02 15:03:24.126373 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126136 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c46f606-8025-4f73-92f9-132f072fb760-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c46f606-8025-4f73-92f9-132f072fb760-tls-certs\") pod \"7c46f606-8025-4f73-92f9-132f072fb760\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " Nov 02 15:03:24.126373 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126169 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c46f606-8025-4f73-92f9-132f072fb760-kube-api-access-6nswq\" (UniqueName: \"kubernetes.io/projected/7c46f606-8025-4f73-92f9-132f072fb760-kube-api-access-6nswq\") pod \"7c46f606-8025-4f73-92f9-132f072fb760\" (UID: \"7c46f606-8025-4f73-92f9-132f072fb760\") " Nov 02 15:03:24.126373 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126303 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-tmp-dir" pod "7c46f606-8025-4f73-92f9-132f072fb760" (UID: "7c46f606-8025-4f73-92f9-132f072fb760"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:03:24.126373 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126340 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-model-cache" pod "7c46f606-8025-4f73-92f9-132f072fb760" (UID: "7c46f606-8025-4f73-92f9-132f072fb760"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:03:24.126645 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126427 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-home" pod "7c46f606-8025-4f73-92f9-132f072fb760" (UID: "7c46f606-8025-4f73-92f9-132f072fb760"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:03:24.126645 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126571 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:24.126645 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126589 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:24.126645 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.126603 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:24.128562 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.128532 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c46f606-8025-4f73-92f9-132f072fb760-tls-certs" pod "7c46f606-8025-4f73-92f9-132f072fb760" (UID: "7c46f606-8025-4f73-92f9-132f072fb760"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:03:24.128709 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.128578 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-dshm" pod "7c46f606-8025-4f73-92f9-132f072fb760" (UID: "7c46f606-8025-4f73-92f9-132f072fb760"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:03:24.128709 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.128609 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c46f606-8025-4f73-92f9-132f072fb760-kube-api-access-6nswq" pod "7c46f606-8025-4f73-92f9-132f072fb760" (UID: "7c46f606-8025-4f73-92f9-132f072fb760"). InnerVolumeSpecName "kube-api-access-6nswq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:03:24.169165 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.169129 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht"] Nov 02 15:03:24.169781 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.169759 5591 cpu_manager.go:414] "containerMap: removing container" podUID="1b74b07e-c864-4cec-85f9-cf1bf0f47cbe" containerName="revoke-keys" Nov 02 15:03:24.169828 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.169786 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b74b07e-c864-4cec-85f9-cf1bf0f47cbe" containerName="revoke-keys" Nov 02 15:03:24.169866 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.169839 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7c46f606-8025-4f73-92f9-132f072fb760" containerName="main" Nov 02 15:03:24.169866 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.169848 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c46f606-8025-4f73-92f9-132f072fb760" containerName="main" Nov 02 15:03:24.169948 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.169936 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c46f606-8025-4f73-92f9-132f072fb760" containerName="main" Nov 02 15:03:24.170013 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.169958 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b74b07e-c864-4cec-85f9-cf1bf0f47cbe" containerName="revoke-keys" Nov 02 15:03:24.173511 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.173488 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.200043 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.195731 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-6e5c6f-openshift-default-dockercfg-tqxlv\"" Nov 02 15:03:24.200043 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.197765 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht"] Nov 02 15:03:24.227712 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.227673 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqlqj\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-kube-api-access-cqlqj\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.227913 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.227774 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-credential-socket\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.227913 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.227870 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-podinfo\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.228064 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228021 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-certs\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.228106 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228070 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-socket\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.228202 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228171 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-token\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.228365 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228220 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-data\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.228365 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228247 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2023392-aa2c-4b54-90c9-80510eb297f0-istiod-ca-cert\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.228365 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228272 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-envoy\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.228549 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228393 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c46f606-8025-4f73-92f9-132f072fb760-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:24.228549 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228410 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c46f606-8025-4f73-92f9-132f072fb760-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:24.228549 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.228423 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nswq\" (UniqueName: \"kubernetes.io/projected/7c46f606-8025-4f73-92f9-132f072fb760-kube-api-access-6nswq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:03:24.291885 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.291849 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:24.329256 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329176 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-podinfo\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329256 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329243 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-certs\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329462 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329269 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-socket\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329462 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329314 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-token\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329462 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329342 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-data\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329462 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329366 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2023392-aa2c-4b54-90c9-80510eb297f0-istiod-ca-cert\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329462 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329390 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-envoy\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329462 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329434 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqlqj\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-kube-api-access-cqlqj\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329781 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329544 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-credential-socket\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329847 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329811 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-socket\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.329916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.329846 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-data\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.330064 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.330027 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-certs\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.330169 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.330113 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-credential-socket\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.330239 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.330219 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2023392-aa2c-4b54-90c9-80510eb297f0-istiod-ca-cert\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.331744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.331724 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-envoy\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.331801 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.331758 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-podinfo\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.369651 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.358719 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqlqj\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-kube-api-access-cqlqj\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.369651 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.361018 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-token\") pod \"e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.454858 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.454819 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:24.489874 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.489844 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:24.592489 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.592409 5591 generic.go:356] "Generic (PLEG): container finished" podID="7c46f606-8025-4f73-92f9-132f072fb760" containerID="b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43" exitCode=0 Nov 02 15:03:24.592489 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.592483 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" Nov 02 15:03:24.592699 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.592483 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" event={"ID":"7c46f606-8025-4f73-92f9-132f072fb760","Type":"ContainerDied","Data":"b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43"} Nov 02 15:03:24.592699 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.592529 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj" event={"ID":"7c46f606-8025-4f73-92f9-132f072fb760","Type":"ContainerDied","Data":"3804d415c6e1485305bbd8c0c83d275369f137fe4a352d614be9bcc56d92689b"} Nov 02 15:03:24.592699 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.592546 5591 scope.go:122] "RemoveContainer" containerID="b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43" Nov 02 15:03:24.601449 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.601427 5591 scope.go:122] "RemoveContainer" containerID="b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43" Nov 02 15:03:24.601719 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:03:24.601699 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43\": container with ID starting with b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43 not found: ID does not exist" containerID="b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43" Nov 02 15:03:24.601774 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.601729 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43"} err="failed to get container status \"b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43\": rpc error: code = NotFound desc = could not find container \"b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43\": container with ID starting with b3d42b0d059c9006ca8fe5d8b0adddf89680b879f632d200bec7dec91d7f9f43 not found: ID does not exist" Nov 02 15:03:24.648195 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.648148 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj"] Nov 02 15:03:24.651882 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.651854 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-5d48887485-gjhbj"] Nov 02 15:03:24.806608 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.806429 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht"] Nov 02 15:03:24.809473 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:03:24.809432 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2023392_aa2c_4b54_90c9_80510eb297f0.slice/crio-47b21505b7df718eac7e4c8da56a56a854d5bf2678741fb6033d08be257424fe WatchSource:0}: Error finding container 47b21505b7df718eac7e4c8da56a56a854d5bf2678741fb6033d08be257424fe: Status 404 returned error can't find the container with id 47b21505b7df718eac7e4c8da56a56a854d5bf2678741fb6033d08be257424fe Nov 02 15:03:24.811622 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.811592 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:03:24.811723 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.811658 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:03:24.811723 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:24.811708 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:03:25.429157 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:25.429120 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c46f606-8025-4f73-92f9-132f072fb760" path="/var/lib/kubelet/pods/7c46f606-8025-4f73-92f9-132f072fb760/volumes" Nov 02 15:03:25.599202 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:25.599158 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" event={"ID":"f2023392-aa2c-4b54-90c9-80510eb297f0","Type":"ContainerStarted","Data":"75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3"} Nov 02 15:03:25.599385 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:25.599218 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" event={"ID":"f2023392-aa2c-4b54-90c9-80510eb297f0","Type":"ContainerStarted","Data":"47b21505b7df718eac7e4c8da56a56a854d5bf2678741fb6033d08be257424fe"} Nov 02 15:03:25.662056 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:25.662000 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podStartSLOduration=1.661967958 podStartE2EDuration="1.661967958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:25.660562019 +0000 UTC m=+4754.386183789" watchObservedRunningTime="2026-11-02 15:03:25.661967958 +0000 UTC m=+4754.387589728" Nov 02 15:03:26.490866 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:26.490833 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:26.495924 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:26.495899 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:26.604139 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:26.604103 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:26.605195 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:26.605175 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:28.190413 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:28.190371 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:28.364648 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:28.364598 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:29.558959 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.558921 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt"] Nov 02 15:03:29.562772 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.562753 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.595682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.595583 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:03:29.595682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.595640 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 15:03:29.595682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.595665 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-6e5c6f-serving-cert\"" Nov 02 15:03:29.596003 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.595733 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:03:29.596003 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.595811 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:03:29.597468 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.597446 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt"] Nov 02 15:03:29.643892 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.643855 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:29.682269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.682231 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.682490 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.682280 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b59529-4eab-4a27-8701-09c3908ca1c2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.682490 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.682378 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkx4\" (UniqueName: \"kubernetes.io/projected/89b59529-4eab-4a27-8701-09c3908ca1c2-kube-api-access-xbkx4\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.682490 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.682427 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.784042 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.783998 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.784221 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.784143 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.784221 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.784190 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b59529-4eab-4a27-8701-09c3908ca1c2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.784320 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.784247 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkx4\" (UniqueName: \"kubernetes.io/projected/89b59529-4eab-4a27-8701-09c3908ca1c2-kube-api-access-xbkx4\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.784869 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.784843 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b59529-4eab-4a27-8701-09c3908ca1c2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.786461 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.786439 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.786562 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.786471 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.813214 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.813097 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:29.813914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.813885 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkx4\" (UniqueName: \"kubernetes.io/projected/89b59529-4eab-4a27-8701-09c3908ca1c2-kube-api-access-xbkx4\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.876020 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.875974 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:29.914618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.914581 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw"] Nov 02 15:03:29.919822 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.919793 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:29.940222 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.940189 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw"] Nov 02 15:03:29.940504 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.940484 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6e5c6f\"" Nov 02 15:03:29.981967 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.981928 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp"] Nov 02 15:03:29.982924 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.982893 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:29.985974 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.985952 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:29.986088 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.986044 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ns8d\" (UniqueName: \"kubernetes.io/projected/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-kube-api-access-4ns8d\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw\" (UID: \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:29.986174 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:29.986099 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw\" (UID: \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:30.004731 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.004701 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp"] Nov 02 15:03:30.008477 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.008441 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a7e7622c-dockercfg-9pb2c\"" Nov 02 15:03:30.087439 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.087343 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ns8d\" (UniqueName: \"kubernetes.io/projected/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-kube-api-access-4ns8d\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw\" (UID: \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:30.088084 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.088054 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw\" (UID: \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:30.089123 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.089103 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2kc\" (UniqueName: \"kubernetes.io/projected/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-kube-api-access-tc2kc\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp\" (UID: \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:30.089305 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.089284 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp\" (UID: \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:30.089398 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.089023 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw\" (UID: \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:30.112928 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.112894 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ns8d\" (UniqueName: \"kubernetes.io/projected/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-kube-api-access-4ns8d\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw\" (UID: \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:30.190010 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.189942 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2kc\" (UniqueName: \"kubernetes.io/projected/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-kube-api-access-tc2kc\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp\" (UID: \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:30.190220 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.190038 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp\" (UID: \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:30.190840 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.190811 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp\" (UID: \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:30.192662 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.192640 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt"] Nov 02 15:03:30.195505 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:03:30.195479 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b59529_4eab_4a27_8701_09c3908ca1c2.slice/crio-691a855aed71af92146fc6fa3208a3dae2f7052dcc0ff147cc17fce2647d5b7f WatchSource:0}: Error finding container 691a855aed71af92146fc6fa3208a3dae2f7052dcc0ff147cc17fce2647d5b7f: Status 404 returned error can't find the container with id 691a855aed71af92146fc6fa3208a3dae2f7052dcc0ff147cc17fce2647d5b7f Nov 02 15:03:30.214813 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.214788 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2kc\" (UniqueName: \"kubernetes.io/projected/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-kube-api-access-tc2kc\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp\" (UID: \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:30.236914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.236883 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:30.299496 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.299277 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:30.543608 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.543571 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw"] Nov 02 15:03:30.547174 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:03:30.547141 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226e4824_108c_4b0b_8ff0_8e6b5f602fc2.slice/crio-9c9b0670328be3258b8b9408e35095566fe38f3a5dee5b446ed7c663c2b55785 WatchSource:0}: Error finding container 9c9b0670328be3258b8b9408e35095566fe38f3a5dee5b446ed7c663c2b55785: Status 404 returned error can't find the container with id 9c9b0670328be3258b8b9408e35095566fe38f3a5dee5b446ed7c663c2b55785 Nov 02 15:03:30.601111 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.601044 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp"] Nov 02 15:03:30.604435 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:03:30.604407 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86ffa3f_0a3c_4a4c_b28e_f1080dd1fddc.slice/crio-ad28a44b8bbbcc8b354e25813e700d23f53f812be9e9533229dc007c0f4921ee WatchSource:0}: Error finding container ad28a44b8bbbcc8b354e25813e700d23f53f812be9e9533229dc007c0f4921ee: Status 404 returned error can't find the container with id ad28a44b8bbbcc8b354e25813e700d23f53f812be9e9533229dc007c0f4921ee Nov 02 15:03:30.631673 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.628119 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" event={"ID":"226e4824-108c-4b0b-8ff0-8e6b5f602fc2","Type":"ContainerStarted","Data":"169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16"} Nov 02 15:03:30.631673 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.628186 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" event={"ID":"226e4824-108c-4b0b-8ff0-8e6b5f602fc2","Type":"ContainerStarted","Data":"9c9b0670328be3258b8b9408e35095566fe38f3a5dee5b446ed7c663c2b55785"} Nov 02 15:03:30.631673 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.628587 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:30.631673 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.629961 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" event={"ID":"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc","Type":"ContainerStarted","Data":"ad28a44b8bbbcc8b354e25813e700d23f53f812be9e9533229dc007c0f4921ee"} Nov 02 15:03:30.631673 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.631644 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" event={"ID":"89b59529-4eab-4a27-8701-09c3908ca1c2","Type":"ContainerStarted","Data":"691a855aed71af92146fc6fa3208a3dae2f7052dcc0ff147cc17fce2647d5b7f"} Nov 02 15:03:30.672261 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:30.672205 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" podStartSLOduration=1.672186821 podStartE2EDuration="1.672186821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:30.671282768 +0000 UTC m=+4759.396904537" watchObservedRunningTime="2026-11-02 15:03:30.672186821 +0000 UTC m=+4759.397808589" Nov 02 15:03:31.072814 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:03:31.072772 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:03:31.528351 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:31.528316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:31.638624 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:31.638584 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" event={"ID":"89b59529-4eab-4a27-8701-09c3908ca1c2","Type":"ContainerStarted","Data":"797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311"} Nov 02 15:03:31.639094 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:31.638647 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:31.640322 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:31.640294 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" event={"ID":"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc","Type":"ContainerStarted","Data":"8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab"} Nov 02 15:03:31.640449 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:31.640428 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:31.684915 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:31.684863 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" podStartSLOduration=2.290570602 podStartE2EDuration="2.684849059s" totalImagesPullingTime="394.278457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:31.683467284 +0000 UTC m=+4760.409089054" watchObservedRunningTime="2026-11-02 15:03:31.684849059 +0000 UTC m=+4760.410470835" Nov 02 15:03:31.729344 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:31.729294 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" podStartSLOduration=2.729282385 podStartE2EDuration="2.729282385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:31.727909932 +0000 UTC m=+4760.453531703" watchObservedRunningTime="2026-11-02 15:03:31.729282385 +0000 UTC m=+4760.454904155" Nov 02 15:03:33.126897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:33.126852 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:33.460334 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:33.460290 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:03:34.842016 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:34.841610 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:36.593422 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:36.593389 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:37.660112 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:37.660077 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:03:38.309355 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:38.309311 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:40.050935 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:40.050899 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:41.629666 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:41.629597 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:41.645687 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:41.645651 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:03:42.646158 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:42.646111 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:03:42.646618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:42.646179 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:03:42.648819 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:42.648794 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:03:43.007424 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.007298 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:43.230594 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.230559 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:43.708574 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.708533 5591 generic.go:356] "Generic (PLEG): container finished" podID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerID="75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3" exitCode=0 Nov 02 15:03:43.709116 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.708582 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" event={"ID":"f2023392-aa2c-4b54-90c9-80510eb297f0","Type":"ContainerDied","Data":"75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3"} Nov 02 15:03:43.709116 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.709105 5591 scope.go:122] "RemoveContainer" containerID="75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3" Nov 02 15:03:43.710320 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.710296 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:03:43.710385 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.710360 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:03:43.710423 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.710397 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:03:43.722947 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.722926 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:43.903665 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:43.903627 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:44.716372 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:44.716330 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" event={"ID":"f2023392-aa2c-4b54-90c9-80510eb297f0","Type":"ContainerStarted","Data":"600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09"} Nov 02 15:03:45.173002 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.172933 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:45.362013 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.358212 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv"] Nov 02 15:03:45.363304 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.363285 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.383873 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.383848 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6e5c6f\"/\"openshift-service-ca.crt\"" Nov 02 15:03:45.384228 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.384214 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6e5c6f\"/\"kube-root-ca.crt\"" Nov 02 15:03:45.386294 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.386264 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv"] Nov 02 15:03:45.457086 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.456347 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6e5c6f\"/\"default-dockercfg-5q8wb\"" Nov 02 15:03:45.457086 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.456483 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6e5c6f\"/\"simulator-single-main-6e5c6f-kserve-self-signed-certs\"" Nov 02 15:03:45.473512 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.473482 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-dshm\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.473652 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.473519 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-tmp-dir\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.473652 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.473600 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlh9s\" (UniqueName: \"kubernetes.io/projected/f40d4f03-a559-43c4-b208-f470632f41fe-kube-api-access-hlh9s\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.473745 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.473688 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-model-cache\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.473745 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.473714 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40d4f03-a559-43c4-b208-f470632f41fe-tls-certs\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.473745 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.473740 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-home\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.490389 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.490361 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:45.490567 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.490547 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 02 15:03:45.490619 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.490592 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 02 15:03:45.574916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.574882 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-dshm\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.575132 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.574924 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-tmp-dir\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.575132 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.574945 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlh9s\" (UniqueName: \"kubernetes.io/projected/f40d4f03-a559-43c4-b208-f470632f41fe-kube-api-access-hlh9s\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.575132 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.574965 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-model-cache\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.575132 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.575003 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40d4f03-a559-43c4-b208-f470632f41fe-tls-certs\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.575132 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.575031 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-home\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.575453 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.575425 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-model-cache\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.575526 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.575456 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-tmp-dir\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.575589 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.575556 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-home\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.577463 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.577444 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-dshm\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.577570 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.577552 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40d4f03-a559-43c4-b208-f470632f41fe-tls-certs\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.606058 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.606021 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlh9s\" (UniqueName: \"kubernetes.io/projected/f40d4f03-a559-43c4-b208-f470632f41fe-kube-api-access-hlh9s\") pod \"simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:45.685916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:45.685876 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:46.018039 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:46.018012 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv"] Nov 02 15:03:46.020070 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:03:46.020040 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf40d4f03_a559_43c4_b208_f470632f41fe.slice/crio-ee8bba285ba14b0cf9bc45ac2ff0e00d3dc11a7df34440bdd419af1f903530db WatchSource:0}: Error finding container ee8bba285ba14b0cf9bc45ac2ff0e00d3dc11a7df34440bdd419af1f903530db: Status 404 returned error can't find the container with id ee8bba285ba14b0cf9bc45ac2ff0e00d3dc11a7df34440bdd419af1f903530db Nov 02 15:03:46.491182 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:46.491145 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 02 15:03:46.491365 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:46.491198 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 02 15:03:46.730783 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:46.730746 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" event={"ID":"f40d4f03-a559-43c4-b208-f470632f41fe","Type":"ContainerStarted","Data":"7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1"} Nov 02 15:03:46.730783 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:46.730788 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" event={"ID":"f40d4f03-a559-43c4-b208-f470632f41fe","Type":"ContainerStarted","Data":"ee8bba285ba14b0cf9bc45ac2ff0e00d3dc11a7df34440bdd419af1f903530db"} Nov 02 15:03:46.781111 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:46.781002 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" podStartSLOduration=1.7809699270000001 podStartE2EDuration="1.780969927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:46.778945985 +0000 UTC m=+4775.504567754" watchObservedRunningTime="2026-11-02 15:03:46.780969927 +0000 UTC m=+4775.506591735" Nov 02 15:03:47.154007 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:47.149972 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:47.491061 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:47.490964 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 02 15:03:47.491061 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:47.491037 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 02 15:03:47.686551 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:47.686487 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:47.721423 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:47.721290 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:03:48.491193 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:48.491155 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 02 15:03:48.491633 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:48.491252 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 02 15:03:48.849114 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:48.847862 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:49.490690 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:49.490648 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 02 15:03:49.490886 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:49.490705 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 02 15:03:50.494912 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:50.494872 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:50.495329 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:50.495240 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:50.496072 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:50.496053 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:03:50.552433 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:50.552394 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:52.246151 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:52.246119 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:53.930808 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:53.928360 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:55.619503 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:55.619456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:55.686867 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:55.686826 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:55.699412 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:55.699387 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:55.808197 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:55.808156 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:03:57.226585 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:57.226500 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:58.831343 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:03:58.831306 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:00.444311 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:00.444271 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:01.072811 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:04:01.072764 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:04:02.126422 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:02.126387 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:03.737775 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:03.737727 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:05.310330 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:05.310295 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:06.450700 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:06.450659 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:04:07.050298 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:07.050261 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:08.646542 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:08.646494 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:10.311357 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:10.311316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:11.844901 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:11.844856 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:12.358437 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:12.358408 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:04:12.381197 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:12.381161 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:04:13.429426 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:13.429390 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:14.934603 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:14.934563 5591 scope.go:122] "RemoveContainer" containerID="2b80a85466e2b595f83daf017cb826cbe49e6d2a7fa9a80355dae37267882baf" Nov 02 15:04:14.943469 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:14.943450 5591 scope.go:122] "RemoveContainer" containerID="f08c1611e0ee3d24e241be8ba550e94e3c4e97925d547c59414a329e38c0e4c8" Nov 02 15:04:15.034165 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:15.034129 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:16.683185 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:16.683146 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:18.330599 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:18.330560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:19.948139 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:19.948095 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:21.548047 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:21.548010 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:23.149084 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:23.148967 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:24.868186 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:24.868157 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:26.438506 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:26.438465 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:28.032451 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:28.032370 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:29.643253 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:29.643215 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:29.811711 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:29.811671 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:30.264595 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:30.264556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:30.622312 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:30.622219 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:30.814527 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:30.814482 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:04:31.074948 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:04:31.074910 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:04:31.307276 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:31.307240 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:32.932511 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:32.932470 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:34.538552 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:34.538516 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:35.030722 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:35.030685 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv"] Nov 02 15:04:35.161187 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:35.161152 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:36.144748 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:36.144687 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:36.513098 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:36.513011 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw"] Nov 02 15:04:36.541779 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:36.541740 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp"] Nov 02 15:04:36.705264 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:36.705188 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:36.870160 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:36.870075 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:37.833796 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:37.833692 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:39.151718 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:39.151681 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt"] Nov 02 15:04:39.152311 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:39.151952 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" podUID="89b59529-4eab-4a27-8701-09c3908ca1c2" containerName="maas-api" containerID="cri-o://797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311" gracePeriod=30 Nov 02 15:04:39.277336 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:39.277303 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:40.407483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.407453 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:04:40.540841 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.540749 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-metrics-certs\") pod \"89b59529-4eab-4a27-8701-09c3908ca1c2\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " Nov 02 15:04:40.541040 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.540873 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b59529-4eab-4a27-8701-09c3908ca1c2-kube-api-access-xbkx4\" (UniqueName: \"kubernetes.io/projected/89b59529-4eab-4a27-8701-09c3908ca1c2-kube-api-access-xbkx4\") pod \"89b59529-4eab-4a27-8701-09c3908ca1c2\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " Nov 02 15:04:40.541040 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.540905 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b59529-4eab-4a27-8701-09c3908ca1c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b59529-4eab-4a27-8701-09c3908ca1c2-openshift-service-ca\") pod \"89b59529-4eab-4a27-8701-09c3908ca1c2\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " Nov 02 15:04:40.541040 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.540924 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-maas-api-tls\") pod \"89b59529-4eab-4a27-8701-09c3908ca1c2\" (UID: \"89b59529-4eab-4a27-8701-09c3908ca1c2\") " Nov 02 15:04:40.541340 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.541306 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b59529-4eab-4a27-8701-09c3908ca1c2-openshift-service-ca" pod "89b59529-4eab-4a27-8701-09c3908ca1c2" (UID: "89b59529-4eab-4a27-8701-09c3908ca1c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:04:40.542925 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.542895 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-metrics-certs" pod "89b59529-4eab-4a27-8701-09c3908ca1c2" (UID: "89b59529-4eab-4a27-8701-09c3908ca1c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:04:40.543071 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.543028 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-maas-api-tls" pod "89b59529-4eab-4a27-8701-09c3908ca1c2" (UID: "89b59529-4eab-4a27-8701-09c3908ca1c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:04:40.543122 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.543071 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b59529-4eab-4a27-8701-09c3908ca1c2-kube-api-access-xbkx4" pod "89b59529-4eab-4a27-8701-09c3908ca1c2" (UID: "89b59529-4eab-4a27-8701-09c3908ca1c2"). InnerVolumeSpecName "kube-api-access-xbkx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:04:40.642011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.641939 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkx4\" (UniqueName: \"kubernetes.io/projected/89b59529-4eab-4a27-8701-09c3908ca1c2-kube-api-access-xbkx4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:40.642011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.641972 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b59529-4eab-4a27-8701-09c3908ca1c2-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:40.642238 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.642034 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:40.642238 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:40.642047 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b59529-4eab-4a27-8701-09c3908ca1c2-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:41.044163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.044125 5591 generic.go:356] "Generic (PLEG): container finished" podID="89b59529-4eab-4a27-8701-09c3908ca1c2" containerID="797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311" exitCode=0 Nov 02 15:04:41.044337 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.044176 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" Nov 02 15:04:41.044337 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.044217 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" event={"ID":"89b59529-4eab-4a27-8701-09c3908ca1c2","Type":"ContainerDied","Data":"797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311"} Nov 02 15:04:41.044337 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.044278 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt" event={"ID":"89b59529-4eab-4a27-8701-09c3908ca1c2","Type":"ContainerDied","Data":"691a855aed71af92146fc6fa3208a3dae2f7052dcc0ff147cc17fce2647d5b7f"} Nov 02 15:04:41.044337 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.044302 5591 scope.go:122] "RemoveContainer" containerID="797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311" Nov 02 15:04:41.053788 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.053768 5591 scope.go:122] "RemoveContainer" containerID="797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311" Nov 02 15:04:41.054225 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:04:41.054200 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311\": container with ID starting with 797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311 not found: ID does not exist" containerID="797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311" Nov 02 15:04:41.054337 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.054231 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311"} err="failed to get container status \"797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311\": rpc error: code = NotFound desc = could not find container \"797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311\": container with ID starting with 797dd95a868162d9c1ce13f2c6e8347167faf5c8ad543bb847d7f7851025a311 not found: ID does not exist" Nov 02 15:04:41.095017 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.094975 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt"] Nov 02 15:04:41.106932 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.106907 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-js9lt"] Nov 02 15:04:41.429269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.429236 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b59529-4eab-4a27-8701-09c3908ca1c2" path="/var/lib/kubelet/pods/89b59529-4eab-4a27-8701-09c3908ca1c2/volumes" Nov 02 15:04:41.508523 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.508477 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:04:41.513867 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.513811 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" podUID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" containerName="payload-processing" containerID="cri-o://169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16" gracePeriod=25 Nov 02 15:04:41.544740 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.544632 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" podUID="f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" containerName="payload-pre-processing" containerID="cri-o://8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab" gracePeriod=25 Nov 02 15:04:41.658121 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:41.657220 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" podUID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:04:42.647665 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:42.647618 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" podUID="f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:04:42.669614 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:42.669556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:42.903835 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:42.903717 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:50.050920 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.050843 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" podUID="f40d4f03-a559-43c4-b208-f470632f41fe" containerName="main" containerID="cri-o://7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1" gracePeriod=45 Nov 02 15:04:50.304574 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.304509 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:04:50.340802 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.340766 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40d4f03-a559-43c4-b208-f470632f41fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/f40d4f03-a559-43c4-b208-f470632f41fe-tls-certs\") pod \"f40d4f03-a559-43c4-b208-f470632f41fe\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " Nov 02 15:04:50.340978 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.340812 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-tmp-dir\") pod \"f40d4f03-a559-43c4-b208-f470632f41fe\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " Nov 02 15:04:50.340978 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.340885 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-home\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-home\") pod \"f40d4f03-a559-43c4-b208-f470632f41fe\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " Nov 02 15:04:50.340978 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.340947 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-model-cache\") pod \"f40d4f03-a559-43c4-b208-f470632f41fe\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " Nov 02 15:04:50.340978 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.340979 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-dshm\") pod \"f40d4f03-a559-43c4-b208-f470632f41fe\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " Nov 02 15:04:50.341282 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.341040 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40d4f03-a559-43c4-b208-f470632f41fe-kube-api-access-hlh9s\" (UniqueName: \"kubernetes.io/projected/f40d4f03-a559-43c4-b208-f470632f41fe-kube-api-access-hlh9s\") pod \"f40d4f03-a559-43c4-b208-f470632f41fe\" (UID: \"f40d4f03-a559-43c4-b208-f470632f41fe\") " Nov 02 15:04:50.341282 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.341179 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-tmp-dir" pod "f40d4f03-a559-43c4-b208-f470632f41fe" (UID: "f40d4f03-a559-43c4-b208-f470632f41fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:50.341282 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.341245 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-home" pod "f40d4f03-a559-43c4-b208-f470632f41fe" (UID: "f40d4f03-a559-43c4-b208-f470632f41fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:50.341452 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.341291 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-model-cache" pod "f40d4f03-a559-43c4-b208-f470632f41fe" (UID: "f40d4f03-a559-43c4-b208-f470632f41fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:50.341452 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.341439 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:50.341548 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.341479 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:50.341548 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.341488 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:50.343483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.343457 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-dshm" pod "f40d4f03-a559-43c4-b208-f470632f41fe" (UID: "f40d4f03-a559-43c4-b208-f470632f41fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:50.343697 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.343669 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40d4f03-a559-43c4-b208-f470632f41fe-tls-certs" pod "f40d4f03-a559-43c4-b208-f470632f41fe" (UID: "f40d4f03-a559-43c4-b208-f470632f41fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:04:50.343782 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.343673 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40d4f03-a559-43c4-b208-f470632f41fe-kube-api-access-hlh9s" pod "f40d4f03-a559-43c4-b208-f470632f41fe" (UID: "f40d4f03-a559-43c4-b208-f470632f41fe"). InnerVolumeSpecName "kube-api-access-hlh9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:04:50.442150 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.442119 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40d4f03-a559-43c4-b208-f470632f41fe-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:50.442150 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.442146 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlh9s\" (UniqueName: \"kubernetes.io/projected/f40d4f03-a559-43c4-b208-f470632f41fe-kube-api-access-hlh9s\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:50.442150 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:50.442158 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40d4f03-a559-43c4-b208-f470632f41fe-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:04:51.094824 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.094788 5591 generic.go:356] "Generic (PLEG): container finished" podID="f40d4f03-a559-43c4-b208-f470632f41fe" containerID="7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1" exitCode=0 Nov 02 15:04:51.095308 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.094833 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" Nov 02 15:04:51.095308 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.094879 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" event={"ID":"f40d4f03-a559-43c4-b208-f470632f41fe","Type":"ContainerDied","Data":"7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1"} Nov 02 15:04:51.095308 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.094926 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv" event={"ID":"f40d4f03-a559-43c4-b208-f470632f41fe","Type":"ContainerDied","Data":"ee8bba285ba14b0cf9bc45ac2ff0e00d3dc11a7df34440bdd419af1f903530db"} Nov 02 15:04:51.095308 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.094944 5591 scope.go:122] "RemoveContainer" containerID="7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1" Nov 02 15:04:51.104269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.104254 5591 scope.go:122] "RemoveContainer" containerID="7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1" Nov 02 15:04:51.104517 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:04:51.104498 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1\": container with ID starting with 7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1 not found: ID does not exist" containerID="7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1" Nov 02 15:04:51.104559 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.104528 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1"} err="failed to get container status \"7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1\": rpc error: code = NotFound desc = could not find container \"7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1\": container with ID starting with 7dc99be9e2a7e8fd1b1e4c91272b58908bc3ee8c5dc41e7c5d6d299ef1787bb1 not found: ID does not exist" Nov 02 15:04:51.145525 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.145493 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv"] Nov 02 15:04:51.157531 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.157501 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-6e5c6f/simulator-single-main-6e5c6f-kserve-686869d9d7-xhcjv"] Nov 02 15:04:51.429499 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.429461 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40d4f03-a559-43c4-b208-f470632f41fe" path="/var/lib/kubelet/pods/f40d4f03-a559-43c4-b208-f470632f41fe/volumes" Nov 02 15:04:51.642005 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:51.641957 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" podUID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:04:52.647085 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:04:52.647041 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" podUID="f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:05:01.073409 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:05:01.073367 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:05:01.641897 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:05:01.641866 5591 logging.go:55] [core] [Channel #18282 SubChannel #18283]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.63:50052", ServerName: "10.133.1.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.63:50052: connect: connection refused" Nov 02 15:05:01.803440 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.803411 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:05:01.822974 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.822949 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:05:01.839756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.839732 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-kube-api-access-4ns8d\" (UniqueName: \"kubernetes.io/projected/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-kube-api-access-4ns8d\") pod \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\" (UID: \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\") " Nov 02 15:05:01.839841 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.839782 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-plugins-config-volume\") pod \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\" (UID: \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\") " Nov 02 15:05:01.839841 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.839806 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-plugins-config-volume\") pod \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\" (UID: \"226e4824-108c-4b0b-8ff0-8e6b5f602fc2\") " Nov 02 15:05:01.839921 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.839865 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-kube-api-access-tc2kc\" (UniqueName: \"kubernetes.io/projected/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-kube-api-access-tc2kc\") pod \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\" (UID: \"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc\") " Nov 02 15:05:01.840232 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.840205 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-plugins-config-volume" pod "f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" (UID: "f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:05:01.840333 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.840207 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-plugins-config-volume" pod "226e4824-108c-4b0b-8ff0-8e6b5f602fc2" (UID: "226e4824-108c-4b0b-8ff0-8e6b5f602fc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:05:01.840333 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.840287 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:01.840333 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.840305 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:01.841784 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.841765 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-kube-api-access-4ns8d" pod "226e4824-108c-4b0b-8ff0-8e6b5f602fc2" (UID: "226e4824-108c-4b0b-8ff0-8e6b5f602fc2"). InnerVolumeSpecName "kube-api-access-4ns8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:01.841928 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.841908 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-kube-api-access-tc2kc" pod "f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" (UID: "f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc"). InnerVolumeSpecName "kube-api-access-tc2kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:01.941133 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.941094 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc2kc\" (UniqueName: \"kubernetes.io/projected/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc-kube-api-access-tc2kc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:01.941133 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:01.941126 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ns8d\" (UniqueName: \"kubernetes.io/projected/226e4824-108c-4b0b-8ff0-8e6b5f602fc2-kube-api-access-4ns8d\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:02.147179 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.147144 5591 generic.go:356] "Generic (PLEG): container finished" podID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" containerID="169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16" exitCode=0 Nov 02 15:05:02.147705 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.147187 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" Nov 02 15:05:02.147705 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.147224 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" event={"ID":"226e4824-108c-4b0b-8ff0-8e6b5f602fc2","Type":"ContainerDied","Data":"169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16"} Nov 02 15:05:02.147705 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.147281 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" event={"ID":"226e4824-108c-4b0b-8ff0-8e6b5f602fc2","Type":"ContainerDied","Data":"9c9b0670328be3258b8b9408e35095566fe38f3a5dee5b446ed7c663c2b55785"} Nov 02 15:05:02.147705 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.147301 5591 scope.go:122] "RemoveContainer" containerID="169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16" Nov 02 15:05:02.148714 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.148684 5591 generic.go:356] "Generic (PLEG): container finished" podID="f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" containerID="8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab" exitCode=0 Nov 02 15:05:02.148853 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.148708 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" event={"ID":"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc","Type":"ContainerDied","Data":"8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab"} Nov 02 15:05:02.148853 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.148768 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" event={"ID":"f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc","Type":"ContainerDied","Data":"ad28a44b8bbbcc8b354e25813e700d23f53f812be9e9533229dc007c0f4921ee"} Nov 02 15:05:02.148853 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.148726 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp" Nov 02 15:05:02.157481 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.157457 5591 scope.go:122] "RemoveContainer" containerID="169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16" Nov 02 15:05:02.157801 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:05:02.157776 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16\": container with ID starting with 169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16 not found: ID does not exist" containerID="169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16" Nov 02 15:05:02.157879 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.157815 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16"} err="failed to get container status \"169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16\": rpc error: code = NotFound desc = could not find container \"169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16\": container with ID starting with 169e378d031613e193a3d87498a931a52258219f96e99ed2f01a4bf8873bbe16 not found: ID does not exist" Nov 02 15:05:02.157879 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.157841 5591 scope.go:122] "RemoveContainer" containerID="8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab" Nov 02 15:05:02.165967 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.165945 5591 scope.go:122] "RemoveContainer" containerID="8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab" Nov 02 15:05:02.166252 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:05:02.166227 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab\": container with ID starting with 8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab not found: ID does not exist" containerID="8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab" Nov 02 15:05:02.166322 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.166263 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab"} err="failed to get container status \"8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab\": rpc error: code = NotFound desc = could not find container \"8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab\": container with ID starting with 8aa6c26b67a0d0e85040185b6e979e17048156545b82b4360653c91fc3529cab not found: ID does not exist" Nov 02 15:05:02.197493 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.197455 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw"] Nov 02 15:05:02.203527 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.203491 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw"] Nov 02 15:05:02.237368 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.237334 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp"] Nov 02 15:05:02.242801 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.242770 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97phtzp"] Nov 02 15:05:02.642062 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:02.642017 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d467775pfdw" podUID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.63:50052\" within 1s: context deadline exceeded" Nov 02 15:05:03.429163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:03.429126 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" path="/var/lib/kubelet/pods/226e4824-108c-4b0b-8ff0-8e6b5f602fc2/volumes" Nov 02 15:05:03.429551 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:03.429534 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" path="/var/lib/kubelet/pods/f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc/volumes" Nov 02 15:05:09.849453 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:09.849413 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:10.026323 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:10.026286 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:11.151841 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.151802 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t"] Nov 02 15:05:11.152269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152207 5591 cpu_manager.go:414] "containerMap: removing container" podUID="89b59529-4eab-4a27-8701-09c3908ca1c2" containerName="maas-api" Nov 02 15:05:11.152269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152219 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b59529-4eab-4a27-8701-09c3908ca1c2" containerName="maas-api" Nov 02 15:05:11.152269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152249 5591 cpu_manager.go:414] "containerMap: removing container" podUID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" containerName="payload-processing" Nov 02 15:05:11.152269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152254 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" containerName="payload-processing" Nov 02 15:05:11.152269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152261 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f40d4f03-a559-43c4-b208-f470632f41fe" containerName="main" Nov 02 15:05:11.152269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152267 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40d4f03-a559-43c4-b208-f470632f41fe" containerName="main" Nov 02 15:05:11.152483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152282 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" containerName="payload-pre-processing" Nov 02 15:05:11.152483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152287 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" containerName="payload-pre-processing" Nov 02 15:05:11.152483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152343 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b59529-4eab-4a27-8701-09c3908ca1c2" containerName="maas-api" Nov 02 15:05:11.152483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152355 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="226e4824-108c-4b0b-8ff0-8e6b5f602fc2" containerName="payload-processing" Nov 02 15:05:11.152483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152362 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40d4f03-a559-43c4-b208-f470632f41fe" containerName="main" Nov 02 15:05:11.152483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.152368 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86ffa3f-0a3c-4a4c-b28e-f1080dd1fddc" containerName="payload-pre-processing" Nov 02 15:05:11.157116 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.157094 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.194175 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.194132 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-6e5c6f-serving-cert\"" Nov 02 15:05:11.194175 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.194151 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 15:05:11.194175 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.194162 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:05:11.194479 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.194222 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:05:11.194479 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.194403 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:05:11.196746 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.196719 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t"] Nov 02 15:05:11.224806 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.224775 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.224979 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.224847 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.224979 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.224943 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlg2c\" (UniqueName: \"kubernetes.io/projected/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-kube-api-access-rlg2c\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.224979 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.224978 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.325907 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.325870 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlg2c\" (UniqueName: \"kubernetes.io/projected/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-kube-api-access-rlg2c\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.326135 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.325916 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.326135 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.325974 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.326135 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.326035 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.326878 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.326849 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.328698 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.328667 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.328823 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.328800 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.330167 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.330141 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:11.416149 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.416067 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlg2c\" (UniqueName: \"kubernetes.io/projected/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-kube-api-access-rlg2c\") pod \"maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.476204 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.476170 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:11.554708 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.554641 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:11.755244 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.755157 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:11.791317 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.791281 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t"] Nov 02 15:05:11.802237 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.802210 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:05:11.925885 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:11.925847 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:12.203894 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:12.203855 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" event={"ID":"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f","Type":"ContainerStarted","Data":"7f8e979dfccf753ead7189f332e76a03b80e0b8c73f38bf0792f00a30d70dc9f"} Nov 02 15:05:13.123973 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:13.123937 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:13.209917 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:13.209880 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" event={"ID":"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f","Type":"ContainerStarted","Data":"3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf"} Nov 02 15:05:13.210381 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:13.209942 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:13.284964 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:13.284907 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" podStartSLOduration=1.862696975 podStartE2EDuration="2.284891884s" totalImagesPullingTime="422.194909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:13.284080332 +0000 UTC m=+4862.009702103" watchObservedRunningTime="2026-11-02 15:05:13.284891884 +0000 UTC m=+4862.010513655" Nov 02 15:05:13.749025 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:13.748962 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:14.732227 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:14.732183 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:14.944517 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:14.944475 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:05:16.532871 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:16.532831 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:18.146716 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:18.146677 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:19.220443 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:19.220412 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:19.811014 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:19.810250 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:19.982668 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:19.982637 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:21.435232 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:21.435199 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:23.043092 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:23.042701 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:24.660528 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:24.660491 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:26.325398 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:26.325354 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:28.028353 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:28.028311 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:29.650226 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:29.650178 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:29.888684 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:29.888647 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:31.072945 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:05:31.072906 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:05:31.333022 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:31.332860 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:32.899854 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:32.899812 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:33.339507 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:33.339428 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:33.508302 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:33.508265 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:34.229584 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.229550 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t"] Nov 02 15:05:34.230097 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.229783 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" podUID="e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" containerName="maas-api" containerID="cri-o://3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf" gracePeriod=30 Nov 02 15:05:34.360328 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.360294 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:34.502052 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.493668 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:34.584688 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.584650 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-kube-api-access-rlg2c\" (UniqueName: \"kubernetes.io/projected/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-kube-api-access-rlg2c\") pod \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " Nov 02 15:05:34.584871 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.584706 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-maas-api-tls\") pod \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " Nov 02 15:05:34.584871 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.584763 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-metrics-certs\") pod \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " Nov 02 15:05:34.584871 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.584808 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-openshift-service-ca\") pod \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\" (UID: \"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f\") " Nov 02 15:05:34.585272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.585240 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-openshift-service-ca" pod "e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" (UID: "e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:05:34.586884 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.586858 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-metrics-certs" pod "e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" (UID: "e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:05:34.586884 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.586871 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-maas-api-tls" pod "e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" (UID: "e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:05:34.587070 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.586953 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-kube-api-access-rlg2c" pod "e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" (UID: "e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f"). InnerVolumeSpecName "kube-api-access-rlg2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:34.686098 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.686051 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:34.686098 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.686088 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:34.686098 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.686102 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:34.686382 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.686116 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlg2c\" (UniqueName: \"kubernetes.io/projected/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f-kube-api-access-rlg2c\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:34.837968 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.837867 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:34.997882 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:34.997847 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:35.299089 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.299053 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht"] Nov 02 15:05:35.299537 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.299319 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" containerID="cri-o://600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09" gracePeriod=30 Nov 02 15:05:35.316456 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.316417 5591 generic.go:356] "Generic (PLEG): container finished" podID="e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" containerID="3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf" exitCode=0 Nov 02 15:05:35.316615 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.316483 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" Nov 02 15:05:35.316615 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.316493 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" event={"ID":"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f","Type":"ContainerDied","Data":"3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf"} Nov 02 15:05:35.316615 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.316550 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t" event={"ID":"e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f","Type":"ContainerDied","Data":"7f8e979dfccf753ead7189f332e76a03b80e0b8c73f38bf0792f00a30d70dc9f"} Nov 02 15:05:35.316615 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.316571 5591 scope.go:122] "RemoveContainer" containerID="3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf" Nov 02 15:05:35.325932 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.325906 5591 scope.go:122] "RemoveContainer" containerID="3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf" Nov 02 15:05:35.326247 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:05:35.326228 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf\": container with ID starting with 3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf not found: ID does not exist" containerID="3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf" Nov 02 15:05:35.326305 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.326255 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf"} err="failed to get container status \"3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf\": rpc error: code = NotFound desc = could not find container \"3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf\": container with ID starting with 3f2faaba15cfbd1b43f74e954fc7e1b4cc3f2a108afbc80463f6faf651bdccdf not found: ID does not exist" Nov 02 15:05:35.418579 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.418539 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:35.419445 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.419409 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t"] Nov 02 15:05:35.421728 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.421700 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6e5c6f-764cbc8fbc-r524t"] Nov 02 15:05:35.445149 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.445113 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" path="/var/lib/kubelet/pods/e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f/volumes" Nov 02 15:05:35.498071 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.498029 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:05:35.498245 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:35.498106 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:05:37.343391 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:37.343346 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:05:40.551109 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.551082 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:05:40.660244 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660213 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-kube-api-access-cqlqj\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-kube-api-access-cqlqj\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660434 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660307 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-envoy\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660434 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660328 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-credential-socket\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660434 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660345 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-data\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660434 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660369 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-socket\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660434 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660394 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-podinfo\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660458 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2023392-aa2c-4b54-90c9-80510eb297f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2023392-aa2c-4b54-90c9-80510eb297f0-istiod-ca-cert\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660538 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-certs\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660567 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-token\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-token\") pod \"f2023392-aa2c-4b54-90c9-80510eb297f0\" (UID: \"f2023392-aa2c-4b54-90c9-80510eb297f0\") " Nov 02 15:05:40.660717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660677 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-data" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:40.660717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660693 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-credential-socket" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:40.660717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660711 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-socket" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:40.661020 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660922 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-certs" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:40.661020 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660936 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:40.661020 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660956 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:40.661020 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.660972 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:40.661198 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.661026 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2023392-aa2c-4b54-90c9-80510eb297f0-istiod-ca-cert" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:05:40.662691 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.662671 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-token" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:40.662761 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.662743 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-podinfo" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:05:40.662964 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.662938 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-envoy" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:40.663099 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.662973 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-kube-api-access-cqlqj" pod "f2023392-aa2c-4b54-90c9-80510eb297f0" (UID: "f2023392-aa2c-4b54-90c9-80510eb297f0"). InnerVolumeSpecName "kube-api-access-cqlqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:40.761756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.761704 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2023392-aa2c-4b54-90c9-80510eb297f0-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:40.761756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.761749 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:40.761756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.761760 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:40.761756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.761769 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqlqj\" (UniqueName: \"kubernetes.io/projected/f2023392-aa2c-4b54-90c9-80510eb297f0-kube-api-access-cqlqj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:40.761756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.761778 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:40.762127 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:40.761786 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2023392-aa2c-4b54-90c9-80510eb297f0-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:05:41.359521 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.359482 5591 generic.go:356] "Generic (PLEG): container finished" podID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerID="600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09" exitCode=0 Nov 02 15:05:41.359744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.359530 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" event={"ID":"f2023392-aa2c-4b54-90c9-80510eb297f0","Type":"ContainerDied","Data":"600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09"} Nov 02 15:05:41.359744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.359603 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" Nov 02 15:05:41.359744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.359620 5591 scope.go:122] "RemoveContainer" containerID="600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09" Nov 02 15:05:41.359744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.359608 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht" event={"ID":"f2023392-aa2c-4b54-90c9-80510eb297f0","Type":"ContainerDied","Data":"47b21505b7df718eac7e4c8da56a56a854d5bf2678741fb6033d08be257424fe"} Nov 02 15:05:41.369574 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.369552 5591 scope.go:122] "RemoveContainer" containerID="75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3" Nov 02 15:05:41.377501 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.377482 5591 scope.go:122] "RemoveContainer" containerID="600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09" Nov 02 15:05:41.377759 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:05:41.377738 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09\": container with ID starting with 600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09 not found: ID does not exist" containerID="600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09" Nov 02 15:05:41.377810 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.377777 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09"} err="failed to get container status \"600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09\": rpc error: code = NotFound desc = could not find container \"600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09\": container with ID starting with 600d66a2e318ef36acdc7781e1974c7b75dfc2be23c4afcaff59dfe19dc5ad09 not found: ID does not exist" Nov 02 15:05:41.377810 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.377796 5591 scope.go:122] "RemoveContainer" containerID="75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3" Nov 02 15:05:41.378047 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:05:41.378026 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3\": container with ID starting with 75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3 not found: ID does not exist" containerID="75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3" Nov 02 15:05:41.378144 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.378047 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3"} err="failed to get container status \"75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3\": rpc error: code = NotFound desc = could not find container \"75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3\": container with ID starting with 75b6de5de2b0458e13d01c1b862295faa59618ca8b4bdf7aeba9d7c858fe71c3 not found: ID does not exist" Nov 02 15:05:41.428969 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.428937 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht"] Nov 02 15:05:41.439306 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:41.439280 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6e5c6f-openshift-default-7975d48577h4vht"] Nov 02 15:05:43.428653 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:43.428615 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" path="/var/lib/kubelet/pods/f2023392-aa2c-4b54-90c9-80510eb297f0/volumes" Nov 02 15:05:50.294154 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.294114 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9"] Nov 02 15:05:50.294562 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.294514 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" Nov 02 15:05:50.294562 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.294530 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" Nov 02 15:05:50.294562 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.294548 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" containerName="maas-api" Nov 02 15:05:50.294562 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.294554 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" containerName="maas-api" Nov 02 15:05:50.294744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.294623 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42abfda-f0f1-4e9c-8c0c-e906ed7feb7f" containerName="maas-api" Nov 02 15:05:50.294744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.294634 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" Nov 02 15:05:50.299101 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.299078 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.327854 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.327823 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7540119b-kserve-self-signed-certs\"" Nov 02 15:05:50.329459 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.329436 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9"] Nov 02 15:05:50.403001 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.402948 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:50.460258 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.460223 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-model-cache\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.460457 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.460273 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-dshm\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.460457 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.460341 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-home\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.460457 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.460414 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-tmp-dir\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.460457 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.460444 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91491693-8296-4acc-a8ab-bc9f7759df77-tls-certs\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.460631 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.460476 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhmq\" (UniqueName: \"kubernetes.io/projected/91491693-8296-4acc-a8ab-bc9f7759df77-kube-api-access-blhmq\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.561522 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.561431 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-dshm\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.561522 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.561474 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-home\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.561783 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.561527 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-tmp-dir\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.561783 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.561567 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91491693-8296-4acc-a8ab-bc9f7759df77-tls-certs\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.561783 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.561602 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhmq\" (UniqueName: \"kubernetes.io/projected/91491693-8296-4acc-a8ab-bc9f7759df77-kube-api-access-blhmq\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.561783 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.561676 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-model-cache\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.562066 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.561957 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-home\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.562066 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.562002 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-tmp-dir\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.562165 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.562071 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-model-cache\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.563758 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.563734 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-dshm\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.564131 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.564113 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91491693-8296-4acc-a8ab-bc9f7759df77-tls-certs\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.584694 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.584663 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhmq\" (UniqueName: \"kubernetes.io/projected/91491693-8296-4acc-a8ab-bc9f7759df77-kube-api-access-blhmq\") pod \"e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.618160 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.618123 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:50.916011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:50.915834 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9"] Nov 02 15:05:50.918822 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:05:50.918793 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91491693_8296_4acc_a8ab_bc9f7759df77.slice/crio-ac384e19c40ef79f621498a8b6cf23358cd4f010f3000ced2a31a47c8fbcb497 WatchSource:0}: Error finding container ac384e19c40ef79f621498a8b6cf23358cd4f010f3000ced2a31a47c8fbcb497: Status 404 returned error can't find the container with id ac384e19c40ef79f621498a8b6cf23358cd4f010f3000ced2a31a47c8fbcb497 Nov 02 15:05:51.409432 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:51.409395 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" event={"ID":"91491693-8296-4acc-a8ab-bc9f7759df77","Type":"ContainerStarted","Data":"f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786"} Nov 02 15:05:51.409432 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:51.409445 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" event={"ID":"91491693-8296-4acc-a8ab-bc9f7759df77","Type":"ContainerStarted","Data":"ac384e19c40ef79f621498a8b6cf23358cd4f010f3000ced2a31a47c8fbcb497"} Nov 02 15:05:51.450728 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:51.450667 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" podStartSLOduration=1.450648937 podStartE2EDuration="1.450648937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:51.450252663 +0000 UTC m=+4900.175874436" watchObservedRunningTime="2026-11-02 15:05:51.450648937 +0000 UTC m=+4900.176270708" Nov 02 15:05:51.618641 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:51.618606 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:05:52.407443 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:05:52.407401 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:06:00.618836 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:00.618796 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:06:00.631466 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:00.631438 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:06:01.073295 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:06:01.073249 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:06:01.470086 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:01.470055 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:06:04.028356 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:04.028320 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c"] Nov 02 15:06:04.040254 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:04.040226 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-65h5c"] Nov 02 15:06:05.429054 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:05.429017 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d4b372-786a-4683-ac87-bd97c991d100" path="/var/lib/kubelet/pods/a0d4b372-786a-4683-ac87-bd97c991d100/volumes" Nov 02 15:06:06.434912 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:06.434880 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:08.626054 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:08.625978 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:06:11.787538 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:11.787395 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:23.218715 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.218676 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l"] Nov 02 15:06:23.219309 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.219285 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" Nov 02 15:06:23.219403 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.219312 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" Nov 02 15:06:23.219475 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.219458 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2023392-aa2c-4b54-90c9-80510eb297f0" containerName="istio-proxy" Nov 02 15:06:23.224025 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.223976 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.253023 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.251381 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:06:23.253023 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.251459 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:06:23.253023 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.251523 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:06:23.253023 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.251555 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:06:23.253676 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.253653 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l"] Nov 02 15:06:23.330349 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.330302 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.330535 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.330438 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.330535 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.330514 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzng\" (UniqueName: \"kubernetes.io/projected/055daa10-1749-4b7f-97b4-6cb59eee93dc-kube-api-access-cvzng\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.432019 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.431969 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzng\" (UniqueName: \"kubernetes.io/projected/055daa10-1749-4b7f-97b4-6cb59eee93dc-kube-api-access-cvzng\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.432210 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.432054 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.432210 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.432106 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.432750 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.432725 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.434394 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.434375 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.455221 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.455194 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzng\" (UniqueName: \"kubernetes.io/projected/055daa10-1749-4b7f-97b4-6cb59eee93dc-kube-api-access-cvzng\") pod \"maas-api-revoke-sub-452b6d7b-6fv6l\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.537458 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.537363 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:23.833574 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:23.833475 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l"] Nov 02 15:06:23.836552 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:06:23.836514 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055daa10_1749_4b7f_97b4_6cb59eee93dc.slice/crio-132f72d56c5ae3a319b0ae02412145b20e0a8d663dffa90b946786bcac26e626 WatchSource:0}: Error finding container 132f72d56c5ae3a319b0ae02412145b20e0a8d663dffa90b946786bcac26e626: Status 404 returned error can't find the container with id 132f72d56c5ae3a319b0ae02412145b20e0a8d663dffa90b946786bcac26e626 Nov 02 15:06:24.516690 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:24.516611 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:06:24.585778 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:24.585734 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" event={"ID":"055daa10-1749-4b7f-97b4-6cb59eee93dc","Type":"ContainerStarted","Data":"5a782203577c50f7dede64148173736bf3eea993dbb050e76df4b09f3fc76898"} Nov 02 15:06:24.585977 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:24.585791 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" event={"ID":"055daa10-1749-4b7f-97b4-6cb59eee93dc","Type":"ContainerStarted","Data":"132f72d56c5ae3a319b0ae02412145b20e0a8d663dffa90b946786bcac26e626"} Nov 02 15:06:24.626417 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:24.626350 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" podStartSLOduration=1.626338097 podStartE2EDuration="1.626338097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:24.625476188 +0000 UTC m=+4933.351097957" watchObservedRunningTime="2026-11-02 15:06:24.626338097 +0000 UTC m=+4933.351959866" Nov 02 15:06:25.594086 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:25.594044 5591 generic.go:356] "Generic (PLEG): container finished" podID="055daa10-1749-4b7f-97b4-6cb59eee93dc" containerID="5a782203577c50f7dede64148173736bf3eea993dbb050e76df4b09f3fc76898" exitCode=0 Nov 02 15:06:25.594490 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:25.594197 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" event={"ID":"055daa10-1749-4b7f-97b4-6cb59eee93dc","Type":"ContainerDied","Data":"5a782203577c50f7dede64148173736bf3eea993dbb050e76df4b09f3fc76898"} Nov 02 15:06:27.019632 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.019595 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:27.117920 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.117873 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055daa10-1749-4b7f-97b4-6cb59eee93dc-kube-api-access-cvzng\" (UniqueName: \"kubernetes.io/projected/055daa10-1749-4b7f-97b4-6cb59eee93dc-kube-api-access-cvzng\") pod \"055daa10-1749-4b7f-97b4-6cb59eee93dc\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " Nov 02 15:06:27.118256 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.118236 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-xks-ca\") pod \"055daa10-1749-4b7f-97b4-6cb59eee93dc\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " Nov 02 15:06:27.118497 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.118478 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-openshift-ca\") pod \"055daa10-1749-4b7f-97b4-6cb59eee93dc\" (UID: \"055daa10-1749-4b7f-97b4-6cb59eee93dc\") " Nov 02 15:06:27.118937 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.118909 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-openshift-ca" pod "055daa10-1749-4b7f-97b4-6cb59eee93dc" (UID: "055daa10-1749-4b7f-97b4-6cb59eee93dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:06:27.120705 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.120681 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-xks-ca" pod "055daa10-1749-4b7f-97b4-6cb59eee93dc" (UID: "055daa10-1749-4b7f-97b4-6cb59eee93dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:06:27.120705 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.120692 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055daa10-1749-4b7f-97b4-6cb59eee93dc-kube-api-access-cvzng" pod "055daa10-1749-4b7f-97b4-6cb59eee93dc" (UID: "055daa10-1749-4b7f-97b4-6cb59eee93dc"). InnerVolumeSpecName "kube-api-access-cvzng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:27.219437 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.219336 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:27.219437 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.219378 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvzng\" (UniqueName: \"kubernetes.io/projected/055daa10-1749-4b7f-97b4-6cb59eee93dc-kube-api-access-cvzng\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:27.219437 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.219389 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055daa10-1749-4b7f-97b4-6cb59eee93dc-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:27.615295 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.615266 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" Nov 02 15:06:27.615473 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.615303 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l" event={"ID":"055daa10-1749-4b7f-97b4-6cb59eee93dc","Type":"ContainerDied","Data":"132f72d56c5ae3a319b0ae02412145b20e0a8d663dffa90b946786bcac26e626"} Nov 02 15:06:27.615473 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.615352 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="132f72d56c5ae3a319b0ae02412145b20e0a8d663dffa90b946786bcac26e626" Nov 02 15:06:27.869502 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.869089 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wrbs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:06:27.869502 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:27.869157 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wrbs7" podUID="dae59a45-652a-46dc-a5e7-c983d14b2b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:06:30.028719 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:30.028675 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9"] Nov 02 15:06:30.031663 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:30.031625 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:30.040710 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:30.040679 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pn2m9"] Nov 02 15:06:30.275749 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:30.275711 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:31.072550 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:06:31.072508 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:06:31.429088 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:31.429050 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925c0f4d-7bb4-4580-8c85-f3b899735d7f" path="/var/lib/kubelet/pods/925c0f4d-7bb4-4580-8c85-f3b899735d7f/volumes" Nov 02 15:06:34.374914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:34.374881 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9"] Nov 02 15:06:34.480952 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:34.480913 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:36.349916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:36.349880 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:06:37.231152 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:37.231114 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:40.846917 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.846872 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w"] Nov 02 15:06:40.847500 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.847475 5591 cpu_manager.go:414] "containerMap: removing container" podUID="055daa10-1749-4b7f-97b4-6cb59eee93dc" containerName="revoke-keys" Nov 02 15:06:40.847592 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.847501 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055daa10-1749-4b7f-97b4-6cb59eee93dc" containerName="revoke-keys" Nov 02 15:06:40.847654 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.847641 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="055daa10-1749-4b7f-97b4-6cb59eee93dc" containerName="revoke-keys" Nov 02 15:06:40.851114 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.851092 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:40.868396 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.868366 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:06:40.880428 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.880398 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:06:40.880584 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.880449 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:06:40.880652 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.880632 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:06:40.882292 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.882273 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w"] Nov 02 15:06:40.953283 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.953244 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:40.953476 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.953313 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:40.953476 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:40.953386 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9dw\" (UniqueName: \"kubernetes.io/projected/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-kube-api-access-fn9dw\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:41.054352 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.054305 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:41.054540 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.054365 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:41.054540 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.054396 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9dw\" (UniqueName: \"kubernetes.io/projected/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-kube-api-access-fn9dw\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:41.055012 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.054975 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:41.056775 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.056755 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:41.077489 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.077462 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9dw\" (UniqueName: \"kubernetes.io/projected/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-kube-api-access-fn9dw\") pod \"maas-api-revoke-sub-3e31a506-4lj4w\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:41.164287 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.164249 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:41.488950 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.488915 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w"] Nov 02 15:06:41.491466 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:06:41.491434 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70bf106_46ab_44c5_94e1_4c1ccc807f9e.slice/crio-0fe6da6c270ec09991ffad3463074347f818658801e8377efecdbe0e7d8fd8b7 WatchSource:0}: Error finding container 0fe6da6c270ec09991ffad3463074347f818658801e8377efecdbe0e7d8fd8b7: Status 404 returned error can't find the container with id 0fe6da6c270ec09991ffad3463074347f818658801e8377efecdbe0e7d8fd8b7 Nov 02 15:06:41.696838 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:41.696789 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" event={"ID":"b70bf106-46ab-44c5-94e1-4c1ccc807f9e","Type":"ContainerStarted","Data":"0fe6da6c270ec09991ffad3463074347f818658801e8377efecdbe0e7d8fd8b7"} Nov 02 15:06:42.708863 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:42.708799 5591 generic.go:356] "Generic (PLEG): container finished" podID="b70bf106-46ab-44c5-94e1-4c1ccc807f9e" containerID="1c6939ab959303bb7e9f155ef7d299fe854513ec68bb3535cd95b76bf02b5197" exitCode=0 Nov 02 15:06:42.709392 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:42.709063 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" event={"ID":"b70bf106-46ab-44c5-94e1-4c1ccc807f9e","Type":"ContainerDied","Data":"1c6939ab959303bb7e9f155ef7d299fe854513ec68bb3535cd95b76bf02b5197"} Nov 02 15:06:43.870314 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:43.870284 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:43.915337 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:43.915286 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-xks-ca\") pod \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " Nov 02 15:06:43.915535 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:43.915473 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-kube-api-access-fn9dw\" (UniqueName: \"kubernetes.io/projected/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-kube-api-access-fn9dw\") pod \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " Nov 02 15:06:43.915535 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:43.915522 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-openshift-ca\") pod \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\" (UID: \"b70bf106-46ab-44c5-94e1-4c1ccc807f9e\") " Nov 02 15:06:43.916091 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:43.916060 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-openshift-ca" pod "b70bf106-46ab-44c5-94e1-4c1ccc807f9e" (UID: "b70bf106-46ab-44c5-94e1-4c1ccc807f9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:06:43.919011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:43.917714 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-kube-api-access-fn9dw" pod "b70bf106-46ab-44c5-94e1-4c1ccc807f9e" (UID: "b70bf106-46ab-44c5-94e1-4c1ccc807f9e"). InnerVolumeSpecName "kube-api-access-fn9dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:43.919011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:43.917858 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-xks-ca" pod "b70bf106-46ab-44c5-94e1-4c1ccc807f9e" (UID: "b70bf106-46ab-44c5-94e1-4c1ccc807f9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:06:44.017317 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:44.017223 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:44.017317 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:44.017264 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:44.017317 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:44.017278 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9dw\" (UniqueName: \"kubernetes.io/projected/b70bf106-46ab-44c5-94e1-4c1ccc807f9e-kube-api-access-fn9dw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:44.706052 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:44.706008 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:06:44.720513 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:44.720477 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" Nov 02 15:06:44.720710 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:44.720483 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w" event={"ID":"b70bf106-46ab-44c5-94e1-4c1ccc807f9e","Type":"ContainerDied","Data":"0fe6da6c270ec09991ffad3463074347f818658801e8377efecdbe0e7d8fd8b7"} Nov 02 15:06:44.720710 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:44.720591 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fe6da6c270ec09991ffad3463074347f818658801e8377efecdbe0e7d8fd8b7" Nov 02 15:06:46.460114 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:46.460065 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:46.621425 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:46.621389 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:47.028374 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:47.028337 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt"] Nov 02 15:06:47.040513 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:47.040480 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6hktt"] Nov 02 15:06:47.429641 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:47.429599 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce753aa7-2155-4bfc-ba8e-b5629124d180" path="/var/lib/kubelet/pods/ce753aa7-2155-4bfc-ba8e-b5629124d180/volumes" Nov 02 15:06:49.396047 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.395978 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" podUID="91491693-8296-4acc-a8ab-bc9f7759df77" containerName="main" containerID="cri-o://f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786" gracePeriod=45 Nov 02 15:06:49.459946 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.459905 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" podUID="91491693-8296-4acc-a8ab-bc9f7759df77" containerName="main" probeResult="failure" output="Get \"https://10.133.1.67:8000/ready\": dial tcp 10.133.1.67:8000: connect: connection refused" Nov 02 15:06:49.641204 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.641173 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:06:49.745299 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.745144 5591 generic.go:356] "Generic (PLEG): container finished" podID="91491693-8296-4acc-a8ab-bc9f7759df77" containerID="f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786" exitCode=0 Nov 02 15:06:49.745299 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.745185 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" Nov 02 15:06:49.745299 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.745221 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" event={"ID":"91491693-8296-4acc-a8ab-bc9f7759df77","Type":"ContainerDied","Data":"f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786"} Nov 02 15:06:49.745299 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.745253 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9" event={"ID":"91491693-8296-4acc-a8ab-bc9f7759df77","Type":"ContainerDied","Data":"ac384e19c40ef79f621498a8b6cf23358cd4f010f3000ced2a31a47c8fbcb497"} Nov 02 15:06:49.745299 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.745275 5591 scope.go:122] "RemoveContainer" containerID="f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786" Nov 02 15:06:49.755659 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.755637 5591 scope.go:122] "RemoveContainer" containerID="f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786" Nov 02 15:06:49.755964 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:06:49.755936 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786\": container with ID starting with f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786 not found: ID does not exist" containerID="f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786" Nov 02 15:06:49.756033 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.755975 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786"} err="failed to get container status \"f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786\": rpc error: code = NotFound desc = could not find container \"f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786\": container with ID starting with f535496bbf6168b3b05ecdba711d4d61d73736b43167bd60fdc23562de4a0786 not found: ID does not exist" Nov 02 15:06:49.768184 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768157 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91491693-8296-4acc-a8ab-bc9f7759df77-kube-api-access-blhmq\" (UniqueName: \"kubernetes.io/projected/91491693-8296-4acc-a8ab-bc9f7759df77-kube-api-access-blhmq\") pod \"91491693-8296-4acc-a8ab-bc9f7759df77\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " Nov 02 15:06:49.768327 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768236 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-model-cache\") pod \"91491693-8296-4acc-a8ab-bc9f7759df77\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " Nov 02 15:06:49.768327 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768322 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-home\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-home\") pod \"91491693-8296-4acc-a8ab-bc9f7759df77\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " Nov 02 15:06:49.768432 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768345 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-tmp-dir\") pod \"91491693-8296-4acc-a8ab-bc9f7759df77\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " Nov 02 15:06:49.768432 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768378 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91491693-8296-4acc-a8ab-bc9f7759df77-tls-certs\" (UniqueName: \"kubernetes.io/secret/91491693-8296-4acc-a8ab-bc9f7759df77-tls-certs\") pod \"91491693-8296-4acc-a8ab-bc9f7759df77\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " Nov 02 15:06:49.768432 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768422 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-dshm\") pod \"91491693-8296-4acc-a8ab-bc9f7759df77\" (UID: \"91491693-8296-4acc-a8ab-bc9f7759df77\") " Nov 02 15:06:49.768614 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768451 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-model-cache" pod "91491693-8296-4acc-a8ab-bc9f7759df77" (UID: "91491693-8296-4acc-a8ab-bc9f7759df77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:06:49.768614 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768594 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-home" pod "91491693-8296-4acc-a8ab-bc9f7759df77" (UID: "91491693-8296-4acc-a8ab-bc9f7759df77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:06:49.768739 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768611 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-tmp-dir" pod "91491693-8296-4acc-a8ab-bc9f7759df77" (UID: "91491693-8296-4acc-a8ab-bc9f7759df77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:06:49.768806 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768752 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:49.768806 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768774 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:49.768806 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.768787 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:49.770308 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.770279 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91491693-8296-4acc-a8ab-bc9f7759df77-kube-api-access-blhmq" pod "91491693-8296-4acc-a8ab-bc9f7759df77" (UID: "91491693-8296-4acc-a8ab-bc9f7759df77"). InnerVolumeSpecName "kube-api-access-blhmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:49.770443 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.770426 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-dshm" pod "91491693-8296-4acc-a8ab-bc9f7759df77" (UID: "91491693-8296-4acc-a8ab-bc9f7759df77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:06:49.770513 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.770439 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91491693-8296-4acc-a8ab-bc9f7759df77-tls-certs" pod "91491693-8296-4acc-a8ab-bc9f7759df77" (UID: "91491693-8296-4acc-a8ab-bc9f7759df77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:06:49.869513 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.869469 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91491693-8296-4acc-a8ab-bc9f7759df77-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:49.869513 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.869506 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91491693-8296-4acc-a8ab-bc9f7759df77-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:49.869513 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:49.869516 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhmq\" (UniqueName: \"kubernetes.io/projected/91491693-8296-4acc-a8ab-bc9f7759df77-kube-api-access-blhmq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:06:50.089880 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:50.089834 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9"] Nov 02 15:06:50.102074 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:50.102028 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7540119b-kserve-6797b799b5-xb9s9"] Nov 02 15:06:51.429876 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:51.429837 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91491693-8296-4acc-a8ab-bc9f7759df77" path="/var/lib/kubelet/pods/91491693-8296-4acc-a8ab-bc9f7759df77/volumes" Nov 02 15:06:52.047611 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:52.047572 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:06:58.029192 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:58.029107 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh"] Nov 02 15:06:58.041201 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:58.041165 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zvvh"] Nov 02 15:06:59.428633 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:06:59.428598 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c" path="/var/lib/kubelet/pods/2b0df51f-70f8-4ca4-ab65-a0dc0201bb7c/volumes" Nov 02 15:07:01.073047 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:07:01.073001 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:07:05.504574 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:05.504539 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:07:09.615296 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.615259 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql"] Nov 02 15:07:09.615703 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.615646 5591 cpu_manager.go:414] "containerMap: removing container" podUID="b70bf106-46ab-44c5-94e1-4c1ccc807f9e" containerName="revoke-keys" Nov 02 15:07:09.615703 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.615658 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70bf106-46ab-44c5-94e1-4c1ccc807f9e" containerName="revoke-keys" Nov 02 15:07:09.615703 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.615675 5591 cpu_manager.go:414] "containerMap: removing container" podUID="91491693-8296-4acc-a8ab-bc9f7759df77" containerName="main" Nov 02 15:07:09.615703 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.615681 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91491693-8296-4acc-a8ab-bc9f7759df77" containerName="main" Nov 02 15:07:09.615859 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.615747 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70bf106-46ab-44c5-94e1-4c1ccc807f9e" containerName="revoke-keys" Nov 02 15:07:09.615859 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.615758 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="91491693-8296-4acc-a8ab-bc9f7759df77" containerName="main" Nov 02 15:07:09.619023 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.619005 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.648099 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.648058 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:07:09.648099 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.648098 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:07:09.648344 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.648153 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:07:09.648344 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.648195 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:07:09.649633 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.649606 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql"] Nov 02 15:07:09.690591 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.690555 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.690793 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.690612 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj22w\" (UniqueName: \"kubernetes.io/projected/e879ae80-6cb5-4fb4-97b4-21141ca898ac-kube-api-access-zj22w\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.690877 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.690856 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.791578 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.791541 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj22w\" (UniqueName: \"kubernetes.io/projected/e879ae80-6cb5-4fb4-97b4-21141ca898ac-kube-api-access-zj22w\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.791731 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.791642 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.791731 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.791676 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.792314 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.792280 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.794062 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.794034 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.814646 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.814619 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj22w\" (UniqueName: \"kubernetes.io/projected/e879ae80-6cb5-4fb4-97b4-21141ca898ac-kube-api-access-zj22w\") pod \"maas-api-revoke-sub-c3929db6-pxqql\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:09.931580 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:09.931553 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:10.237844 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:10.237818 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql"] Nov 02 15:07:10.240484 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:07:10.240458 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode879ae80_6cb5_4fb4_97b4_21141ca898ac.slice/crio-8648b614e317069a87fb9cbc377ef8be0de96b37d099f3b83af9beec3c57d4a3 WatchSource:0}: Error finding container 8648b614e317069a87fb9cbc377ef8be0de96b37d099f3b83af9beec3c57d4a3: Status 404 returned error can't find the container with id 8648b614e317069a87fb9cbc377ef8be0de96b37d099f3b83af9beec3c57d4a3 Nov 02 15:07:10.933681 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:10.933645 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" event={"ID":"e879ae80-6cb5-4fb4-97b4-21141ca898ac","Type":"ContainerStarted","Data":"d6d6d56eea0f3b8e54bb7b4c43c1fc116672e77ee8be2dd2a1c348d58c797d7a"} Nov 02 15:07:10.934109 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:10.933701 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" event={"ID":"e879ae80-6cb5-4fb4-97b4-21141ca898ac","Type":"ContainerStarted","Data":"8648b614e317069a87fb9cbc377ef8be0de96b37d099f3b83af9beec3c57d4a3"} Nov 02 15:07:10.974903 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:10.974849 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" podStartSLOduration=1.9748359519999998 podStartE2EDuration="1.974835952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:10.974194446 +0000 UTC m=+4979.699816216" watchObservedRunningTime="2026-11-02 15:07:10.974835952 +0000 UTC m=+4979.700457722" Nov 02 15:07:11.939570 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:11.939535 5591 generic.go:356] "Generic (PLEG): container finished" podID="e879ae80-6cb5-4fb4-97b4-21141ca898ac" containerID="d6d6d56eea0f3b8e54bb7b4c43c1fc116672e77ee8be2dd2a1c348d58c797d7a" exitCode=0 Nov 02 15:07:11.939976 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:11.939626 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" event={"ID":"e879ae80-6cb5-4fb4-97b4-21141ca898ac","Type":"ContainerDied","Data":"d6d6d56eea0f3b8e54bb7b4c43c1fc116672e77ee8be2dd2a1c348d58c797d7a"} Nov 02 15:07:13.073090 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.073058 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:13.223285 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.223195 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-xks-ca\") pod \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " Nov 02 15:07:13.223285 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.223269 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-openshift-ca\") pod \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " Nov 02 15:07:13.223522 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.223330 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e879ae80-6cb5-4fb4-97b4-21141ca898ac-kube-api-access-zj22w\" (UniqueName: \"kubernetes.io/projected/e879ae80-6cb5-4fb4-97b4-21141ca898ac-kube-api-access-zj22w\") pod \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\" (UID: \"e879ae80-6cb5-4fb4-97b4-21141ca898ac\") " Nov 02 15:07:13.223656 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.223624 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-openshift-ca" pod "e879ae80-6cb5-4fb4-97b4-21141ca898ac" (UID: "e879ae80-6cb5-4fb4-97b4-21141ca898ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:07:13.225460 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.225436 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-xks-ca" pod "e879ae80-6cb5-4fb4-97b4-21141ca898ac" (UID: "e879ae80-6cb5-4fb4-97b4-21141ca898ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:13.225563 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.225437 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e879ae80-6cb5-4fb4-97b4-21141ca898ac-kube-api-access-zj22w" pod "e879ae80-6cb5-4fb4-97b4-21141ca898ac" (UID: "e879ae80-6cb5-4fb4-97b4-21141ca898ac"). InnerVolumeSpecName "kube-api-access-zj22w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:13.324279 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.324244 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj22w\" (UniqueName: \"kubernetes.io/projected/e879ae80-6cb5-4fb4-97b4-21141ca898ac-kube-api-access-zj22w\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:07:13.324279 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.324277 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:07:13.324488 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.324290 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e879ae80-6cb5-4fb4-97b4-21141ca898ac-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:07:13.956944 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.956913 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" Nov 02 15:07:13.957150 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.956905 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql" event={"ID":"e879ae80-6cb5-4fb4-97b4-21141ca898ac","Type":"ContainerDied","Data":"8648b614e317069a87fb9cbc377ef8be0de96b37d099f3b83af9beec3c57d4a3"} Nov 02 15:07:13.957150 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:13.957072 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8648b614e317069a87fb9cbc377ef8be0de96b37d099f3b83af9beec3c57d4a3" Nov 02 15:07:15.014097 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:15.014063 5591 scope.go:122] "RemoveContainer" containerID="06f9e2eba48c0160eaf50276d41cbc0ba2eccc5822d90b0748d6995d79eb7dcf" Nov 02 15:07:20.839496 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:20.839453 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:07:31.073259 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:07:31.073214 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:07:32.379736 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:32.379689 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:32.544248 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:32.544211 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:35.102335 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.102294 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8"] Nov 02 15:07:35.102773 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.102710 5591 cpu_manager.go:414] "containerMap: removing container" podUID="e879ae80-6cb5-4fb4-97b4-21141ca898ac" containerName="revoke-keys" Nov 02 15:07:35.102773 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.102722 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e879ae80-6cb5-4fb4-97b4-21141ca898ac" containerName="revoke-keys" Nov 02 15:07:35.102866 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.102801 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e879ae80-6cb5-4fb4-97b4-21141ca898ac" containerName="revoke-keys" Nov 02 15:07:35.105876 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.105856 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:35.140665 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.140631 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8"] Nov 02 15:07:35.223449 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.223407 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc668bcd-ffac-4b86-9511-4938be404cd3-plugins-config-volume\") pod \"payload-pre-processing-79fd4bfffc-jmhb8\" (UID: \"bc668bcd-ffac-4b86-9511-4938be404cd3\") " pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:35.223639 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.223468 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbhw\" (UniqueName: \"kubernetes.io/projected/bc668bcd-ffac-4b86-9511-4938be404cd3-kube-api-access-dnbhw\") pod \"payload-pre-processing-79fd4bfffc-jmhb8\" (UID: \"bc668bcd-ffac-4b86-9511-4938be404cd3\") " pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:35.324801 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.324747 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc668bcd-ffac-4b86-9511-4938be404cd3-plugins-config-volume\") pod \"payload-pre-processing-79fd4bfffc-jmhb8\" (UID: \"bc668bcd-ffac-4b86-9511-4938be404cd3\") " pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:35.325025 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.324826 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbhw\" (UniqueName: \"kubernetes.io/projected/bc668bcd-ffac-4b86-9511-4938be404cd3-kube-api-access-dnbhw\") pod \"payload-pre-processing-79fd4bfffc-jmhb8\" (UID: \"bc668bcd-ffac-4b86-9511-4938be404cd3\") " pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:35.325544 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.325521 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc668bcd-ffac-4b86-9511-4938be404cd3-plugins-config-volume\") pod \"payload-pre-processing-79fd4bfffc-jmhb8\" (UID: \"bc668bcd-ffac-4b86-9511-4938be404cd3\") " pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:35.348769 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.348736 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbhw\" (UniqueName: \"kubernetes.io/projected/bc668bcd-ffac-4b86-9511-4938be404cd3-kube-api-access-dnbhw\") pod \"payload-pre-processing-79fd4bfffc-jmhb8\" (UID: \"bc668bcd-ffac-4b86-9511-4938be404cd3\") " pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:35.418882 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.418849 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:35.718151 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:35.718124 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8"] Nov 02 15:07:35.720959 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:07:35.720927 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc668bcd_ffac_4b86_9511_4938be404cd3.slice/crio-a5700fcfa5ac05f1a2df63f1734c8022320974cb03e1462dc6bae6bee194e89b WatchSource:0}: Error finding container a5700fcfa5ac05f1a2df63f1734c8022320974cb03e1462dc6bae6bee194e89b: Status 404 returned error can't find the container with id a5700fcfa5ac05f1a2df63f1734c8022320974cb03e1462dc6bae6bee194e89b Nov 02 15:07:36.067734 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:36.067631 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" event={"ID":"bc668bcd-ffac-4b86-9511-4938be404cd3","Type":"ContainerStarted","Data":"04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6"} Nov 02 15:07:36.067734 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:36.067699 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" event={"ID":"bc668bcd-ffac-4b86-9511-4938be404cd3","Type":"ContainerStarted","Data":"a5700fcfa5ac05f1a2df63f1734c8022320974cb03e1462dc6bae6bee194e89b"} Nov 02 15:07:36.067953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:36.067811 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:36.121201 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:36.121151 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" podStartSLOduration=1.121137962 podStartE2EDuration="1.121137962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:36.120685381 +0000 UTC m=+5004.846307151" watchObservedRunningTime="2026-11-02 15:07:36.121137962 +0000 UTC m=+5004.846759732" Nov 02 15:07:47.075622 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:47.075589 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:07:47.146591 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:47.146554 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9"] Nov 02 15:07:48.788108 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.788069 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-89f975dd6-nhpnw"] Nov 02 15:07:48.791901 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.791881 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:48.811530 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.811500 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-89f975dd6-nhpnw"] Nov 02 15:07:48.851772 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.851736 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce68f58-827d-492e-baa5-a0340f12eb03-plugins-config-volume\") pod \"payload-processing-89f975dd6-nhpnw\" (UID: \"6ce68f58-827d-492e-baa5-a0340f12eb03\") " pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:48.851965 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.851804 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh88x\" (UniqueName: \"kubernetes.io/projected/6ce68f58-827d-492e-baa5-a0340f12eb03-kube-api-access-nh88x\") pod \"payload-processing-89f975dd6-nhpnw\" (UID: \"6ce68f58-827d-492e-baa5-a0340f12eb03\") " pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:48.952970 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.952935 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce68f58-827d-492e-baa5-a0340f12eb03-plugins-config-volume\") pod \"payload-processing-89f975dd6-nhpnw\" (UID: \"6ce68f58-827d-492e-baa5-a0340f12eb03\") " pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:48.953156 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.953019 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh88x\" (UniqueName: \"kubernetes.io/projected/6ce68f58-827d-492e-baa5-a0340f12eb03-kube-api-access-nh88x\") pod \"payload-processing-89f975dd6-nhpnw\" (UID: \"6ce68f58-827d-492e-baa5-a0340f12eb03\") " pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:48.953729 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.953708 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce68f58-827d-492e-baa5-a0340f12eb03-plugins-config-volume\") pod \"payload-processing-89f975dd6-nhpnw\" (UID: \"6ce68f58-827d-492e-baa5-a0340f12eb03\") " pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:48.976402 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:48.976375 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh88x\" (UniqueName: \"kubernetes.io/projected/6ce68f58-827d-492e-baa5-a0340f12eb03-kube-api-access-nh88x\") pod \"payload-processing-89f975dd6-nhpnw\" (UID: \"6ce68f58-827d-492e-baa5-a0340f12eb03\") " pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:49.104935 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:49.104848 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:49.402346 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:49.402302 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-89f975dd6-nhpnw"] Nov 02 15:07:49.405433 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:07:49.405401 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce68f58_827d_492e_baa5_a0340f12eb03.slice/crio-fa38c988f2150b1e762e8968fb7a08753ffab8a4f2340f9995b5a54bee7f6b76 WatchSource:0}: Error finding container fa38c988f2150b1e762e8968fb7a08753ffab8a4f2340f9995b5a54bee7f6b76: Status 404 returned error can't find the container with id fa38c988f2150b1e762e8968fb7a08753ffab8a4f2340f9995b5a54bee7f6b76 Nov 02 15:07:50.132896 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:50.132858 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" event={"ID":"6ce68f58-827d-492e-baa5-a0340f12eb03","Type":"ContainerStarted","Data":"96a79fbc49d32007e99afd48c667dff70253e0948404ff27cb1ab5172633dc2a"} Nov 02 15:07:50.133339 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:50.132915 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" event={"ID":"6ce68f58-827d-492e-baa5-a0340f12eb03","Type":"ContainerStarted","Data":"fa38c988f2150b1e762e8968fb7a08753ffab8a4f2340f9995b5a54bee7f6b76"} Nov 02 15:07:50.133339 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:50.132935 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:07:50.186845 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:50.186789 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" podStartSLOduration=2.186775236 podStartE2EDuration="2.186775236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:50.185686886 +0000 UTC m=+5018.911308655" watchObservedRunningTime="2026-11-02 15:07:50.186775236 +0000 UTC m=+5018.912397012" Nov 02 15:07:52.147614 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:52.147574 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" podUID="91a47106-23e2-439f-abdb-622b606887c4" containerName="payload-pre-processing" containerID="cri-o://43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6" gracePeriod=25 Nov 02 15:07:52.398007 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:52.397937 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 15:07:52.485535 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:52.485505 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a47106-23e2-439f-abdb-622b606887c4-kube-api-access-zn2wf\" (UniqueName: \"kubernetes.io/projected/91a47106-23e2-439f-abdb-622b606887c4-kube-api-access-zn2wf\") pod \"91a47106-23e2-439f-abdb-622b606887c4\" (UID: \"91a47106-23e2-439f-abdb-622b606887c4\") " Nov 02 15:07:52.485713 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:52.485662 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a47106-23e2-439f-abdb-622b606887c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a47106-23e2-439f-abdb-622b606887c4-plugins-config-volume\") pod \"91a47106-23e2-439f-abdb-622b606887c4\" (UID: \"91a47106-23e2-439f-abdb-622b606887c4\") " Nov 02 15:07:52.486075 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:52.486052 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a47106-23e2-439f-abdb-622b606887c4-plugins-config-volume" pod "91a47106-23e2-439f-abdb-622b606887c4" (UID: "91a47106-23e2-439f-abdb-622b606887c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:07:52.487590 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:52.487565 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a47106-23e2-439f-abdb-622b606887c4-kube-api-access-zn2wf" pod "91a47106-23e2-439f-abdb-622b606887c4" (UID: "91a47106-23e2-439f-abdb-622b606887c4"). InnerVolumeSpecName "kube-api-access-zn2wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:52.586325 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:52.586294 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2wf\" (UniqueName: \"kubernetes.io/projected/91a47106-23e2-439f-abdb-622b606887c4-kube-api-access-zn2wf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:07:52.586325 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:52.586322 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a47106-23e2-439f-abdb-622b606887c4-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:07:53.147207 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.147167 5591 generic.go:356] "Generic (PLEG): container finished" podID="91a47106-23e2-439f-abdb-622b606887c4" containerID="43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6" exitCode=0 Nov 02 15:07:53.147413 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.147227 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" event={"ID":"91a47106-23e2-439f-abdb-622b606887c4","Type":"ContainerDied","Data":"43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6"} Nov 02 15:07:53.147413 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.147233 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" Nov 02 15:07:53.147413 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.147292 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9" event={"ID":"91a47106-23e2-439f-abdb-622b606887c4","Type":"ContainerDied","Data":"def028f2b5fcd75814286526c7d57e9dec1f51626bd08a35e67bc253908c4201"} Nov 02 15:07:53.147413 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.147313 5591 scope.go:122] "RemoveContainer" containerID="43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6" Nov 02 15:07:53.155812 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.155789 5591 scope.go:122] "RemoveContainer" containerID="43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6" Nov 02 15:07:53.156169 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:07:53.156111 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6\": container with ID starting with 43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6 not found: ID does not exist" containerID="43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6" Nov 02 15:07:53.156169 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.156139 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6"} err="failed to get container status \"43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6\": rpc error: code = NotFound desc = could not find container \"43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6\": container with ID starting with 43842b1a940b1b83b14b1ab41135a04eb6b5cd22ba4b45854be78c9fc7a7bdf6 not found: ID does not exist" Nov 02 15:07:53.198956 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.198923 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9"] Nov 02 15:07:53.211071 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.211042 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-cxzs9"] Nov 02 15:07:53.428175 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:07:53.428093 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a47106-23e2-439f-abdb-622b606887c4" path="/var/lib/kubelet/pods/91a47106-23e2-439f-abdb-622b606887c4/volumes" Nov 02 15:08:01.028342 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:01.028297 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m"] Nov 02 15:08:01.040098 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:01.040065 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45072ecc-j9s7m"] Nov 02 15:08:01.075522 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:08:01.075486 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:08:01.140746 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:01.140708 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:08:01.210046 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:01.210008 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-t792x"] Nov 02 15:08:01.429003 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:01.428944 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68874cb9-a420-45c2-a1b1-f332d4494a16" path="/var/lib/kubelet/pods/68874cb9-a420-45c2-a1b1-f332d4494a16/volumes" Nov 02 15:08:03.131851 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.131806 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq"] Nov 02 15:08:03.132301 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.132240 5591 cpu_manager.go:414] "containerMap: removing container" podUID="91a47106-23e2-439f-abdb-622b606887c4" containerName="payload-pre-processing" Nov 02 15:08:03.132301 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.132252 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a47106-23e2-439f-abdb-622b606887c4" containerName="payload-pre-processing" Nov 02 15:08:03.132398 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.132310 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a47106-23e2-439f-abdb-622b606887c4" containerName="payload-pre-processing" Nov 02 15:08:03.135381 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.135354 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:03.154659 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.154630 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq"] Nov 02 15:08:03.289798 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.289763 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88a9ba7-3c6d-47dd-aae0-da3b997533e8-plugins-config-volume\") pod \"payload-pre-processing-84dc7c97f-5mmbq\" (UID: \"f88a9ba7-3c6d-47dd-aae0-da3b997533e8\") " pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:03.290032 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.289829 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d6fl\" (UniqueName: \"kubernetes.io/projected/f88a9ba7-3c6d-47dd-aae0-da3b997533e8-kube-api-access-5d6fl\") pod \"payload-pre-processing-84dc7c97f-5mmbq\" (UID: \"f88a9ba7-3c6d-47dd-aae0-da3b997533e8\") " pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:03.390972 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.390884 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88a9ba7-3c6d-47dd-aae0-da3b997533e8-plugins-config-volume\") pod \"payload-pre-processing-84dc7c97f-5mmbq\" (UID: \"f88a9ba7-3c6d-47dd-aae0-da3b997533e8\") " pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:03.391191 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.391041 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6fl\" (UniqueName: \"kubernetes.io/projected/f88a9ba7-3c6d-47dd-aae0-da3b997533e8-kube-api-access-5d6fl\") pod \"payload-pre-processing-84dc7c97f-5mmbq\" (UID: \"f88a9ba7-3c6d-47dd-aae0-da3b997533e8\") " pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:03.391719 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.391696 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88a9ba7-3c6d-47dd-aae0-da3b997533e8-plugins-config-volume\") pod \"payload-pre-processing-84dc7c97f-5mmbq\" (UID: \"f88a9ba7-3c6d-47dd-aae0-da3b997533e8\") " pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:03.414855 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.414822 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6fl\" (UniqueName: \"kubernetes.io/projected/f88a9ba7-3c6d-47dd-aae0-da3b997533e8-kube-api-access-5d6fl\") pod \"payload-pre-processing-84dc7c97f-5mmbq\" (UID: \"f88a9ba7-3c6d-47dd-aae0-da3b997533e8\") " pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:03.447919 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.447885 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:03.749834 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:03.749755 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq"] Nov 02 15:08:03.753114 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:08:03.753083 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88a9ba7_3c6d_47dd_aae0_da3b997533e8.slice/crio-e176a32dd9b347f6fd1d3245eec55d75175c64eb4c137c27259fc64e0ef0b263 WatchSource:0}: Error finding container e176a32dd9b347f6fd1d3245eec55d75175c64eb4c137c27259fc64e0ef0b263: Status 404 returned error can't find the container with id e176a32dd9b347f6fd1d3245eec55d75175c64eb4c137c27259fc64e0ef0b263 Nov 02 15:08:04.203169 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:04.203126 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" event={"ID":"f88a9ba7-3c6d-47dd-aae0-da3b997533e8","Type":"ContainerStarted","Data":"00920c0d80c8d582174345011afb523fe6987f40dceb99d11fac36573c60e13f"} Nov 02 15:08:04.203605 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:04.203193 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" event={"ID":"f88a9ba7-3c6d-47dd-aae0-da3b997533e8","Type":"ContainerStarted","Data":"e176a32dd9b347f6fd1d3245eec55d75175c64eb4c137c27259fc64e0ef0b263"} Nov 02 15:08:04.203605 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:04.203233 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:04.255928 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:04.255866 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" podStartSLOduration=1.255849575 podStartE2EDuration="1.255849575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:04.242687941 +0000 UTC m=+5032.968309711" watchObservedRunningTime="2026-11-02 15:08:04.255849575 +0000 UTC m=+5032.981471344" Nov 02 15:08:06.210357 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:06.210315 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" podUID="566435d0-5ce9-4ef4-9a61-9ebb69b8d17d" containerName="payload-processing" containerID="cri-o://9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d" gracePeriod=25 Nov 02 15:08:06.464526 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:06.464459 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 15:08:06.621238 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:06.621201 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-plugins-config-volume\") pod \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\" (UID: \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\") " Nov 02 15:08:06.621454 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:06.621342 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-kube-api-access-h5gr6\" (UniqueName: \"kubernetes.io/projected/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-kube-api-access-h5gr6\") pod \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\" (UID: \"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d\") " Nov 02 15:08:06.621641 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:06.621618 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-plugins-config-volume" pod "566435d0-5ce9-4ef4-9a61-9ebb69b8d17d" (UID: "566435d0-5ce9-4ef4-9a61-9ebb69b8d17d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:08:06.623599 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:06.623578 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-kube-api-access-h5gr6" pod "566435d0-5ce9-4ef4-9a61-9ebb69b8d17d" (UID: "566435d0-5ce9-4ef4-9a61-9ebb69b8d17d"). InnerVolumeSpecName "kube-api-access-h5gr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:08:06.722494 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:06.722410 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5gr6\" (UniqueName: \"kubernetes.io/projected/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-kube-api-access-h5gr6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:08:06.722494 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:06.722447 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:08:07.219515 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.219475 5591 generic.go:356] "Generic (PLEG): container finished" podID="566435d0-5ce9-4ef4-9a61-9ebb69b8d17d" containerID="9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d" exitCode=0 Nov 02 15:08:07.219969 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.219526 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" Nov 02 15:08:07.219969 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.219563 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" event={"ID":"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d","Type":"ContainerDied","Data":"9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d"} Nov 02 15:08:07.219969 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.219617 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-t792x" event={"ID":"566435d0-5ce9-4ef4-9a61-9ebb69b8d17d","Type":"ContainerDied","Data":"3b6b5c7d12682f29109152c3176a03fc87913c28ced9df920491cdd504c5c4aa"} Nov 02 15:08:07.219969 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.219640 5591 scope.go:122] "RemoveContainer" containerID="9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d" Nov 02 15:08:07.228696 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.228674 5591 scope.go:122] "RemoveContainer" containerID="9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d" Nov 02 15:08:07.228972 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:08:07.228952 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d\": container with ID starting with 9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d not found: ID does not exist" containerID="9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d" Nov 02 15:08:07.229050 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.228999 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d"} err="failed to get container status \"9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d\": rpc error: code = NotFound desc = could not find container \"9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d\": container with ID starting with 9e87235173585c59a658257f38d8c38653246cbf1c9b5630741b5ce40190f44d not found: ID does not exist" Nov 02 15:08:07.270049 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.270014 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-t792x"] Nov 02 15:08:07.282228 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.282199 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-t792x"] Nov 02 15:08:07.429487 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:07.429448 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566435d0-5ce9-4ef4-9a61-9ebb69b8d17d" path="/var/lib/kubelet/pods/566435d0-5ce9-4ef4-9a61-9ebb69b8d17d/volumes" Nov 02 15:08:15.029138 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.029099 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw"] Nov 02 15:08:15.038128 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.038097 5591 scope.go:122] "RemoveContainer" containerID="7914eb45270e81e15ea4f8b8f013293fb83a895a3fbfb2c3f330f66ed120bd2e" Nov 02 15:08:15.041200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.041176 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e058e0-mbntw"] Nov 02 15:08:15.047089 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.047065 5591 scope.go:122] "RemoveContainer" containerID="d697cd379063f41824145bf21d03e1f9202499585d25b350dacbc15ea54d3c4d" Nov 02 15:08:15.054687 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.054663 5591 scope.go:122] "RemoveContainer" containerID="57d5ad75f8aa56ded02d9860842ae685c80c8536d2833f8a215a41ca5b9c9319" Nov 02 15:08:15.126596 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.126567 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-89f975dd6-nhpnw"] Nov 02 15:08:15.210532 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.210503 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-84dc7c97f-5mmbq" Nov 02 15:08:15.278220 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.278181 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8"] Nov 02 15:08:15.431450 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:15.431411 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b74b07e-c864-4cec-85f9-cf1bf0f47cbe" path="/var/lib/kubelet/pods/1b74b07e-c864-4cec-85f9-cf1bf0f47cbe/volumes" Nov 02 15:08:17.393952 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:17.393918 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:17.581396 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:17.581358 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:18.172029 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.171979 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:18.373905 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.373868 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct"] Nov 02 15:08:18.374378 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.374357 5591 cpu_manager.go:414] "containerMap: removing container" podUID="566435d0-5ce9-4ef4-9a61-9ebb69b8d17d" containerName="payload-processing" Nov 02 15:08:18.374485 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.374379 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566435d0-5ce9-4ef4-9a61-9ebb69b8d17d" containerName="payload-processing" Nov 02 15:08:18.374552 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.374512 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="566435d0-5ce9-4ef4-9a61-9ebb69b8d17d" containerName="payload-processing" Nov 02 15:08:18.379231 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.379209 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.395515 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.395162 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 15:08:18.407395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.407355 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 15:08:18.407395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.407363 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 15:08:18.407596 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.407529 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 02 15:08:18.407596 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.407548 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-w2sbh\"" Nov 02 15:08:18.407596 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.407554 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 15:08:18.410092 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.410067 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct"] Nov 02 15:08:18.430481 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.430132 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6b29\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-kube-api-access-p6b29\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.430481 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.430198 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.430481 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.430285 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.430481 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.430318 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.531777 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.531726 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.532237 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.531792 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.532237 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.531857 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6b29\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-kube-api-access-p6b29\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.532237 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.531906 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.532945 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.532913 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.533149 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.532918 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.537922 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.537890 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.555464 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.555430 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6b29\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-kube-api-access-p6b29\") pod \"payload-processing-external-model-5bcd7997-nz5ct\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.692180 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.692084 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:18.992244 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:18.992183 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct"] Nov 02 15:08:18.994737 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:08:18.994704 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5337aa9c_4e6f_47c6_8410_10cdcf49fa16.slice/crio-1b5f1af27686155d8b93f859310f6ed3216183bf18f6cce2073b4a737f4ac5fc WatchSource:0}: Error finding container 1b5f1af27686155d8b93f859310f6ed3216183bf18f6cce2073b4a737f4ac5fc: Status 404 returned error can't find the container with id 1b5f1af27686155d8b93f859310f6ed3216183bf18f6cce2073b4a737f4ac5fc Nov 02 15:08:19.283585 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:19.283498 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" event={"ID":"5337aa9c-4e6f-47c6-8410-10cdcf49fa16","Type":"ContainerStarted","Data":"d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9"} Nov 02 15:08:19.283585 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:19.283552 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" event={"ID":"5337aa9c-4e6f-47c6-8410-10cdcf49fa16","Type":"ContainerStarted","Data":"1b5f1af27686155d8b93f859310f6ed3216183bf18f6cce2073b4a737f4ac5fc"} Nov 02 15:08:19.283789 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:19.283598 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:19.338402 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:19.338348 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" podStartSLOduration=1.3383347620000001 podStartE2EDuration="1.338334762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:19.337070814 +0000 UTC m=+5048.062692583" watchObservedRunningTime="2026-11-02 15:08:19.338334762 +0000 UTC m=+5048.063956553" Nov 02 15:08:19.621205 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:19.621125 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:08:20.127894 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.127842 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" podUID="6ce68f58-827d-492e-baa5-a0340f12eb03" containerName="payload-processing" containerID="cri-o://96a79fbc49d32007e99afd48c667dff70253e0948404ff27cb1ab5172633dc2a" gracePeriod=25 Nov 02 15:08:20.279056 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.278995 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" podUID="bc668bcd-ffac-4b86-9511-4938be404cd3" containerName="payload-pre-processing" containerID="cri-o://04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6" gracePeriod=25 Nov 02 15:08:20.296256 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.296213 5591 generic.go:356] "Generic (PLEG): container finished" podID="6ce68f58-827d-492e-baa5-a0340f12eb03" containerID="96a79fbc49d32007e99afd48c667dff70253e0948404ff27cb1ab5172633dc2a" exitCode=0 Nov 02 15:08:20.297114 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.297081 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" event={"ID":"6ce68f58-827d-492e-baa5-a0340f12eb03","Type":"ContainerDied","Data":"96a79fbc49d32007e99afd48c667dff70253e0948404ff27cb1ab5172633dc2a"} Nov 02 15:08:20.404057 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.404026 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:08:20.467044 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.467008 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ce68f58-827d-492e-baa5-a0340f12eb03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce68f58-827d-492e-baa5-a0340f12eb03-plugins-config-volume\") pod \"6ce68f58-827d-492e-baa5-a0340f12eb03\" (UID: \"6ce68f58-827d-492e-baa5-a0340f12eb03\") " Nov 02 15:08:20.467227 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.467088 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce68f58-827d-492e-baa5-a0340f12eb03-kube-api-access-nh88x\" (UniqueName: \"kubernetes.io/projected/6ce68f58-827d-492e-baa5-a0340f12eb03-kube-api-access-nh88x\") pod \"6ce68f58-827d-492e-baa5-a0340f12eb03\" (UID: \"6ce68f58-827d-492e-baa5-a0340f12eb03\") " Nov 02 15:08:20.467820 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.467789 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce68f58-827d-492e-baa5-a0340f12eb03-plugins-config-volume" pod "6ce68f58-827d-492e-baa5-a0340f12eb03" (UID: "6ce68f58-827d-492e-baa5-a0340f12eb03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:08:20.469574 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.469548 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce68f58-827d-492e-baa5-a0340f12eb03-kube-api-access-nh88x" pod "6ce68f58-827d-492e-baa5-a0340f12eb03" (UID: "6ce68f58-827d-492e-baa5-a0340f12eb03"). InnerVolumeSpecName "kube-api-access-nh88x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:08:20.568754 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.568723 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce68f58-827d-492e-baa5-a0340f12eb03-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:08:20.568754 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.568753 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh88x\" (UniqueName: \"kubernetes.io/projected/6ce68f58-827d-492e-baa5-a0340f12eb03-kube-api-access-nh88x\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:08:20.576687 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.576660 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:08:20.669422 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.669321 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc668bcd-ffac-4b86-9511-4938be404cd3-kube-api-access-dnbhw\" (UniqueName: \"kubernetes.io/projected/bc668bcd-ffac-4b86-9511-4938be404cd3-kube-api-access-dnbhw\") pod \"bc668bcd-ffac-4b86-9511-4938be404cd3\" (UID: \"bc668bcd-ffac-4b86-9511-4938be404cd3\") " Nov 02 15:08:20.669931 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.669424 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc668bcd-ffac-4b86-9511-4938be404cd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc668bcd-ffac-4b86-9511-4938be404cd3-plugins-config-volume\") pod \"bc668bcd-ffac-4b86-9511-4938be404cd3\" (UID: \"bc668bcd-ffac-4b86-9511-4938be404cd3\") " Nov 02 15:08:20.669931 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.669909 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc668bcd-ffac-4b86-9511-4938be404cd3-plugins-config-volume" pod "bc668bcd-ffac-4b86-9511-4938be404cd3" (UID: "bc668bcd-ffac-4b86-9511-4938be404cd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:08:20.672269 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.672237 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc668bcd-ffac-4b86-9511-4938be404cd3-kube-api-access-dnbhw" pod "bc668bcd-ffac-4b86-9511-4938be404cd3" (UID: "bc668bcd-ffac-4b86-9511-4938be404cd3"). InnerVolumeSpecName "kube-api-access-dnbhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:08:20.771039 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.770974 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc668bcd-ffac-4b86-9511-4938be404cd3-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:08:20.771039 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:20.771035 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnbhw\" (UniqueName: \"kubernetes.io/projected/bc668bcd-ffac-4b86-9511-4938be404cd3-kube-api-access-dnbhw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:08:21.306768 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.306726 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" event={"ID":"6ce68f58-827d-492e-baa5-a0340f12eb03","Type":"ContainerDied","Data":"fa38c988f2150b1e762e8968fb7a08753ffab8a4f2340f9995b5a54bee7f6b76"} Nov 02 15:08:21.307098 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.307076 5591 scope.go:122] "RemoveContainer" containerID="96a79fbc49d32007e99afd48c667dff70253e0948404ff27cb1ab5172633dc2a" Nov 02 15:08:21.307412 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.307392 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-89f975dd6-nhpnw" Nov 02 15:08:21.310762 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.310605 5591 generic.go:356] "Generic (PLEG): container finished" podID="bc668bcd-ffac-4b86-9511-4938be404cd3" containerID="04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6" exitCode=0 Nov 02 15:08:21.310762 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.310644 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" event={"ID":"bc668bcd-ffac-4b86-9511-4938be404cd3","Type":"ContainerDied","Data":"04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6"} Nov 02 15:08:21.310762 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.310671 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" event={"ID":"bc668bcd-ffac-4b86-9511-4938be404cd3","Type":"ContainerDied","Data":"a5700fcfa5ac05f1a2df63f1734c8022320974cb03e1462dc6bae6bee194e89b"} Nov 02 15:08:21.310762 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.310719 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8" Nov 02 15:08:21.319390 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.319358 5591 scope.go:122] "RemoveContainer" containerID="04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6" Nov 02 15:08:21.327320 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.327301 5591 scope.go:122] "RemoveContainer" containerID="04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6" Nov 02 15:08:21.327623 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:08:21.327602 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6\": container with ID starting with 04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6 not found: ID does not exist" containerID="04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6" Nov 02 15:08:21.327672 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.327633 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6"} err="failed to get container status \"04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6\": rpc error: code = NotFound desc = could not find container \"04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6\": container with ID starting with 04fd68a263a2d934e07d978630187c170e65ce8506e599e7af82598960e8c7c6 not found: ID does not exist" Nov 02 15:08:21.377548 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.377494 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-89f975dd6-nhpnw"] Nov 02 15:08:21.396087 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.396044 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-89f975dd6-nhpnw"] Nov 02 15:08:21.428515 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.428480 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce68f58-827d-492e-baa5-a0340f12eb03" path="/var/lib/kubelet/pods/6ce68f58-827d-492e-baa5-a0340f12eb03/volumes" Nov 02 15:08:21.435009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.432268 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8"] Nov 02 15:08:21.446004 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:21.437571 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-79fd4bfffc-jmhb8"] Nov 02 15:08:22.549008 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:22.546346 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:08:22.549008 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:22.546428 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:08:23.428180 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:23.428147 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc668bcd-ffac-4b86-9511-4938be404cd3" path="/var/lib/kubelet/pods/bc668bcd-ffac-4b86-9511-4938be404cd3/volumes" Nov 02 15:08:30.217445 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:30.217394 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:30.300259 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:30.300227 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:08:31.073400 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:08:31.073357 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:08:38.618708 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:38.618671 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:49.034787 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:08:49.034750 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:01.073715 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:09:01.073675 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:09:04.078043 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:04.077975 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:04.257497 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:04.257456 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:12.388747 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:12.388713 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:09:12.412358 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:12.412327 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:09:15.086709 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:15.086674 5591 scope.go:122] "RemoveContainer" containerID="5a3888785df1fea42b49dd56368588ac6cab075ab900f1cb5bd20eddcbc57633" Nov 02 15:09:15.095152 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:15.095120 5591 scope.go:122] "RemoveContainer" containerID="c215c3a38d634d70562dd9bf08bfef63d4456b561eebe9f3fbe6d630fefb8785" Nov 02 15:09:30.344726 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:30.344690 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:31.073169 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:09:31.073125 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:09:33.454884 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:33.454844 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:33.614436 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:33.614406 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:56.401953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.401870 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg"] Nov 02 15:09:56.402498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.402263 5591 cpu_manager.go:414] "containerMap: removing container" podUID="6ce68f58-827d-492e-baa5-a0340f12eb03" containerName="payload-processing" Nov 02 15:09:56.402498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.402276 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce68f58-827d-492e-baa5-a0340f12eb03" containerName="payload-processing" Nov 02 15:09:56.402498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.402287 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bc668bcd-ffac-4b86-9511-4938be404cd3" containerName="payload-pre-processing" Nov 02 15:09:56.402498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.402292 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc668bcd-ffac-4b86-9511-4938be404cd3" containerName="payload-pre-processing" Nov 02 15:09:56.402498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.402372 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce68f58-827d-492e-baa5-a0340f12eb03" containerName="payload-processing" Nov 02 15:09:56.402498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.402380 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc668bcd-ffac-4b86-9511-4938be404cd3" containerName="payload-pre-processing" Nov 02 15:09:56.405514 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.405498 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.422889 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.422862 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:09:56.423093 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.422889 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:09:56.434966 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.434940 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:09:56.435136 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.435077 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:09:56.436600 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.436567 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg"] Nov 02 15:09:56.485128 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.485094 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtm2d\" (UniqueName: \"kubernetes.io/projected/d119eef7-761c-49a1-8295-1b4b3406ceee-kube-api-access-xtm2d\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.485279 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.485193 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.485279 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.485220 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.586187 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.586147 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.586396 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.586196 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.586396 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.586292 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtm2d\" (UniqueName: \"kubernetes.io/projected/d119eef7-761c-49a1-8295-1b4b3406ceee-kube-api-access-xtm2d\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.586852 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.586828 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.588621 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.588597 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.609750 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.609710 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtm2d\" (UniqueName: \"kubernetes.io/projected/d119eef7-761c-49a1-8295-1b4b3406ceee-kube-api-access-xtm2d\") pod \"maas-api-revoke-sub-e2547622-pnghg\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:56.718926 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:56.718839 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:09:57.016266 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:57.016241 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg"] Nov 02 15:09:57.018994 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:09:57.018943 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd119eef7_761c_49a1_8295_1b4b3406ceee.slice/crio-0a2c3e5bae64505dcbd9d4acce32b47af1e2637c36c95ffd6c5ec10fe5c70d07 WatchSource:0}: Error finding container 0a2c3e5bae64505dcbd9d4acce32b47af1e2637c36c95ffd6c5ec10fe5c70d07: Status 404 returned error can't find the container with id 0a2c3e5bae64505dcbd9d4acce32b47af1e2637c36c95ffd6c5ec10fe5c70d07 Nov 02 15:09:57.749350 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:57.749313 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" event={"ID":"d119eef7-761c-49a1-8295-1b4b3406ceee","Type":"ContainerStarted","Data":"0b701894b032af665c4d54ff8ad1b2c10aed8aa1494bef3d2746ac795d53dfa4"} Nov 02 15:09:57.749770 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:57.749367 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" event={"ID":"d119eef7-761c-49a1-8295-1b4b3406ceee","Type":"ContainerStarted","Data":"0a2c3e5bae64505dcbd9d4acce32b47af1e2637c36c95ffd6c5ec10fe5c70d07"} Nov 02 15:09:57.790856 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:57.790800 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" podStartSLOduration=1.790787684 podStartE2EDuration="1.790787684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:57.789152056 +0000 UTC m=+5146.514773826" watchObservedRunningTime="2026-11-02 15:09:57.790787684 +0000 UTC m=+5146.516409447" Nov 02 15:09:58.755216 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:58.755180 5591 generic.go:356] "Generic (PLEG): container finished" podID="d119eef7-761c-49a1-8295-1b4b3406ceee" containerID="0b701894b032af665c4d54ff8ad1b2c10aed8aa1494bef3d2746ac795d53dfa4" exitCode=0 Nov 02 15:09:58.755640 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:58.755271 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" event={"ID":"d119eef7-761c-49a1-8295-1b4b3406ceee","Type":"ContainerDied","Data":"0b701894b032af665c4d54ff8ad1b2c10aed8aa1494bef3d2746ac795d53dfa4"} Nov 02 15:09:59.899716 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:09:59.899688 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:10:00.018421 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.018320 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-openshift-ca\") pod \"d119eef7-761c-49a1-8295-1b4b3406ceee\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " Nov 02 15:10:00.018586 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.018447 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-xks-ca\") pod \"d119eef7-761c-49a1-8295-1b4b3406ceee\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " Nov 02 15:10:00.018586 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.018493 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d119eef7-761c-49a1-8295-1b4b3406ceee-kube-api-access-xtm2d\" (UniqueName: \"kubernetes.io/projected/d119eef7-761c-49a1-8295-1b4b3406ceee-kube-api-access-xtm2d\") pod \"d119eef7-761c-49a1-8295-1b4b3406ceee\" (UID: \"d119eef7-761c-49a1-8295-1b4b3406ceee\") " Nov 02 15:10:00.018725 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.018700 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-openshift-ca" pod "d119eef7-761c-49a1-8295-1b4b3406ceee" (UID: "d119eef7-761c-49a1-8295-1b4b3406ceee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:10:00.020586 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.020563 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d119eef7-761c-49a1-8295-1b4b3406ceee-kube-api-access-xtm2d" pod "d119eef7-761c-49a1-8295-1b4b3406ceee" (UID: "d119eef7-761c-49a1-8295-1b4b3406ceee"). InnerVolumeSpecName "kube-api-access-xtm2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:00.020586 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.020560 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-xks-ca" pod "d119eef7-761c-49a1-8295-1b4b3406ceee" (UID: "d119eef7-761c-49a1-8295-1b4b3406ceee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:10:00.119656 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.119609 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtm2d\" (UniqueName: \"kubernetes.io/projected/d119eef7-761c-49a1-8295-1b4b3406ceee-kube-api-access-xtm2d\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:00.119656 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.119647 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:00.119656 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.119661 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d119eef7-761c-49a1-8295-1b4b3406ceee-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:00.766400 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.766363 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" event={"ID":"d119eef7-761c-49a1-8295-1b4b3406ceee","Type":"ContainerDied","Data":"0a2c3e5bae64505dcbd9d4acce32b47af1e2637c36c95ffd6c5ec10fe5c70d07"} Nov 02 15:10:00.766400 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.766385 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg" Nov 02 15:10:00.766634 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:00.766411 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a2c3e5bae64505dcbd9d4acce32b47af1e2637c36c95ffd6c5ec10fe5c70d07" Nov 02 15:10:01.072898 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:10:01.072815 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:10:07.165273 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:07.165232 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct"] Nov 02 15:10:07.304779 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:07.304744 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:07.485045 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:07.484932 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:07.651332 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:07.651298 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:08.539449 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:08.539406 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:10:12.166361 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.166291 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" podUID="5337aa9c-4e6f-47c6-8410-10cdcf49fa16" containerName="payload-processing" containerID="cri-o://d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9" gracePeriod=25 Nov 02 15:10:12.416423 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.416358 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:10:12.435776 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.435745 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-kube-api-access-p6b29\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-kube-api-access-p6b29\") pod \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " Nov 02 15:10:12.435935 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.435832 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-provider-credentials\") pod \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " Nov 02 15:10:12.435935 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.435929 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-plugins-config-volume\") pod \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " Nov 02 15:10:12.436108 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.435970 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-routing-overlay-volume\") pod \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\" (UID: \"5337aa9c-4e6f-47c6-8410-10cdcf49fa16\") " Nov 02 15:10:12.436394 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.436362 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-plugins-config-volume" pod "5337aa9c-4e6f-47c6-8410-10cdcf49fa16" (UID: "5337aa9c-4e6f-47c6-8410-10cdcf49fa16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:10:12.436602 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.436561 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-routing-overlay-volume" pod "5337aa9c-4e6f-47c6-8410-10cdcf49fa16" (UID: "5337aa9c-4e6f-47c6-8410-10cdcf49fa16"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:10:12.438065 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.438043 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-kube-api-access-p6b29" pod "5337aa9c-4e6f-47c6-8410-10cdcf49fa16" (UID: "5337aa9c-4e6f-47c6-8410-10cdcf49fa16"). InnerVolumeSpecName "kube-api-access-p6b29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:12.438398 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.438383 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-provider-credentials" pod "5337aa9c-4e6f-47c6-8410-10cdcf49fa16" (UID: "5337aa9c-4e6f-47c6-8410-10cdcf49fa16"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:12.537238 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.537201 5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:12.537238 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.537240 5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-routing-overlay-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:12.537452 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.537258 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6b29\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-kube-api-access-p6b29\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:12.537452 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.537274 5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5337aa9c-4e6f-47c6-8410-10cdcf49fa16-provider-credentials\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:12.825643 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.825556 5591 generic.go:356] "Generic (PLEG): container finished" podID="5337aa9c-4e6f-47c6-8410-10cdcf49fa16" containerID="d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9" exitCode=0 Nov 02 15:10:12.825643 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.825607 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" Nov 02 15:10:12.825643 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.825619 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" event={"ID":"5337aa9c-4e6f-47c6-8410-10cdcf49fa16","Type":"ContainerDied","Data":"d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9"} Nov 02 15:10:12.825931 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.825660 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct" event={"ID":"5337aa9c-4e6f-47c6-8410-10cdcf49fa16","Type":"ContainerDied","Data":"1b5f1af27686155d8b93f859310f6ed3216183bf18f6cce2073b4a737f4ac5fc"} Nov 02 15:10:12.825931 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.825680 5591 scope.go:122] "RemoveContainer" containerID="d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9" Nov 02 15:10:12.834329 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.834309 5591 scope.go:122] "RemoveContainer" containerID="d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9" Nov 02 15:10:12.834643 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:10:12.834621 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9\": container with ID starting with d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9 not found: ID does not exist" containerID="d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9" Nov 02 15:10:12.834717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.834652 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9"} err="failed to get container status \"d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9\": rpc error: code = NotFound desc = could not find container \"d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9\": container with ID starting with d3695f910049d3d287707a9657657e7654ebd0142c70951f6fec300155884eb9 not found: ID does not exist" Nov 02 15:10:12.876481 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.876455 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct"] Nov 02 15:10:12.888549 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:12.888518 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-nz5ct"] Nov 02 15:10:13.428956 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:13.428917 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5337aa9c-4e6f-47c6-8410-10cdcf49fa16" path="/var/lib/kubelet/pods/5337aa9c-4e6f-47c6-8410-10cdcf49fa16/volumes" Nov 02 15:10:13.931330 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:13.931295 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:10:17.608367 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:17.608324 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:17.773355 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:17.773309 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:18.571887 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.571848 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp"] Nov 02 15:10:18.572293 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.572278 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5337aa9c-4e6f-47c6-8410-10cdcf49fa16" containerName="payload-processing" Nov 02 15:10:18.572346 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.572294 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5337aa9c-4e6f-47c6-8410-10cdcf49fa16" containerName="payload-processing" Nov 02 15:10:18.572346 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.572323 5591 cpu_manager.go:414] "containerMap: removing container" podUID="d119eef7-761c-49a1-8295-1b4b3406ceee" containerName="revoke-keys" Nov 02 15:10:18.572346 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.572329 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d119eef7-761c-49a1-8295-1b4b3406ceee" containerName="revoke-keys" Nov 02 15:10:18.572449 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.572389 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d119eef7-761c-49a1-8295-1b4b3406ceee" containerName="revoke-keys" Nov 02 15:10:18.572449 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.572405 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5337aa9c-4e6f-47c6-8410-10cdcf49fa16" containerName="payload-processing" Nov 02 15:10:18.576973 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.576955 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.595184 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.595155 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:10:18.607312 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.607274 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:10:18.607312 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.607275 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:10:18.607627 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.607598 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:10:18.609267 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.609244 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp"] Nov 02 15:10:18.619440 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.619403 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.619651 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.619560 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.619745 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.619696 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbl7\" (UniqueName: \"kubernetes.io/projected/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-kube-api-access-jdbl7\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.720735 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.720693 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbl7\" (UniqueName: \"kubernetes.io/projected/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-kube-api-access-jdbl7\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.720956 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.720782 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.720956 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.720836 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.721607 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.721574 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.723328 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.723308 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.743929 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.743897 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbl7\" (UniqueName: \"kubernetes.io/projected/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-kube-api-access-jdbl7\") pod \"maas-api-revoke-sub-c1e271a9-vpmdp\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:18.890581 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:18.890553 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:19.188671 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:19.188642 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp"] Nov 02 15:10:19.191437 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:10:19.191406 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ced1d0c_07b3_420c_bf3d_0b8e5ed7bef2.slice/crio-5ef3bb33d59689210fcaf338becedb14f52ba11713a08958093ae7b0437286ed WatchSource:0}: Error finding container 5ef3bb33d59689210fcaf338becedb14f52ba11713a08958093ae7b0437286ed: Status 404 returned error can't find the container with id 5ef3bb33d59689210fcaf338becedb14f52ba11713a08958093ae7b0437286ed Nov 02 15:10:19.891712 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:19.891675 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" event={"ID":"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2","Type":"ContainerStarted","Data":"526fcd54d3d54eb65a98eae34b9f3081d8f61670677fff1c898e367558c7584a"} Nov 02 15:10:19.892144 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:19.891731 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" event={"ID":"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2","Type":"ContainerStarted","Data":"5ef3bb33d59689210fcaf338becedb14f52ba11713a08958093ae7b0437286ed"} Nov 02 15:10:19.932413 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:19.932361 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" podStartSLOduration=1.932346806 podStartE2EDuration="1.932346806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:19.93083237 +0000 UTC m=+5168.656454139" watchObservedRunningTime="2026-11-02 15:10:19.932346806 +0000 UTC m=+5168.657968576" Nov 02 15:10:20.898302 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:20.898266 5591 generic.go:356] "Generic (PLEG): container finished" podID="7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2" containerID="526fcd54d3d54eb65a98eae34b9f3081d8f61670677fff1c898e367558c7584a" exitCode=0 Nov 02 15:10:20.898784 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:20.898352 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" event={"ID":"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2","Type":"ContainerDied","Data":"526fcd54d3d54eb65a98eae34b9f3081d8f61670677fff1c898e367558c7584a"} Nov 02 15:10:22.034897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.034867 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:22.153162 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.153065 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-kube-api-access-jdbl7\" (UniqueName: \"kubernetes.io/projected/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-kube-api-access-jdbl7\") pod \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " Nov 02 15:10:22.153162 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.153147 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-openshift-ca\") pod \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " Nov 02 15:10:22.153409 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.153195 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-xks-ca\") pod \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\" (UID: \"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2\") " Nov 02 15:10:22.153525 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.153502 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-openshift-ca" pod "7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2" (UID: "7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:10:22.155273 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.155241 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-xks-ca" pod "7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2" (UID: "7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:10:22.155407 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.155311 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-kube-api-access-jdbl7" pod "7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2" (UID: "7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2"). InnerVolumeSpecName "kube-api-access-jdbl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:22.254866 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.254830 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:22.254866 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.254861 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:22.254866 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.254871 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbl7\" (UniqueName: \"kubernetes.io/projected/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2-kube-api-access-jdbl7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:22.914372 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.914338 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" Nov 02 15:10:22.914540 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.914348 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp" event={"ID":"7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2","Type":"ContainerDied","Data":"5ef3bb33d59689210fcaf338becedb14f52ba11713a08958093ae7b0437286ed"} Nov 02 15:10:22.914540 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:22.914489 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ef3bb33d59689210fcaf338becedb14f52ba11713a08958093ae7b0437286ed" Nov 02 15:10:29.851765 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:29.851729 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:10:30.440902 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:30.440869 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:31.072768 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:10:31.072723 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:10:34.543523 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:34.543484 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:10:38.509381 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.509344 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4"] Nov 02 15:10:38.509787 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.509744 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2" containerName="revoke-keys" Nov 02 15:10:38.509787 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.509757 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2" containerName="revoke-keys" Nov 02 15:10:38.509866 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.509844 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2" containerName="revoke-keys" Nov 02 15:10:38.513200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.513175 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.528097 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.528072 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:10:38.528253 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.528106 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:10:38.540138 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.540106 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:10:38.540332 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.540307 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:10:38.542274 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.542245 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4"] Nov 02 15:10:38.625894 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.625856 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f945e8bb-6227-4304-a101-961d33363966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.626111 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.625923 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f945e8bb-6227-4304-a101-961d33363966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.626111 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.626088 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47ms\" (UniqueName: \"kubernetes.io/projected/f945e8bb-6227-4304-a101-961d33363966-kube-api-access-w47ms\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.727163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.727126 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47ms\" (UniqueName: \"kubernetes.io/projected/f945e8bb-6227-4304-a101-961d33363966-kube-api-access-w47ms\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.727356 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.727192 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f945e8bb-6227-4304-a101-961d33363966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.727356 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.727239 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f945e8bb-6227-4304-a101-961d33363966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.727867 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.727844 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f945e8bb-6227-4304-a101-961d33363966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.729622 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.729596 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f945e8bb-6227-4304-a101-961d33363966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.750953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.750924 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47ms\" (UniqueName: \"kubernetes.io/projected/f945e8bb-6227-4304-a101-961d33363966-kube-api-access-w47ms\") pod \"maas-api-revoke-sub-2e1f5b5f-9rzj4\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:38.827567 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:38.827491 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:39.130083 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:39.130054 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4"] Nov 02 15:10:39.133241 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:10:39.133214 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf945e8bb_6227_4304_a101_961d33363966.slice/crio-8b2e6a2ec22d802b23ec4bdaab66632d49b682ba1d722a0197269f2291f21307 WatchSource:0}: Error finding container 8b2e6a2ec22d802b23ec4bdaab66632d49b682ba1d722a0197269f2291f21307: Status 404 returned error can't find the container with id 8b2e6a2ec22d802b23ec4bdaab66632d49b682ba1d722a0197269f2291f21307 Nov 02 15:10:40.009905 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:40.009861 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" event={"ID":"f945e8bb-6227-4304-a101-961d33363966","Type":"ContainerStarted","Data":"6896fa1157fdfe9714720f43bab4b12e9ff424234fcf42870b0e0f6c1463a00e"} Nov 02 15:10:40.010350 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:40.009919 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" event={"ID":"f945e8bb-6227-4304-a101-961d33363966","Type":"ContainerStarted","Data":"8b2e6a2ec22d802b23ec4bdaab66632d49b682ba1d722a0197269f2291f21307"} Nov 02 15:10:40.051345 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:40.051295 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" podStartSLOduration=2.051280674 podStartE2EDuration="2.051280674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:40.049541704 +0000 UTC m=+5188.775163475" watchObservedRunningTime="2026-11-02 15:10:40.051280674 +0000 UTC m=+5188.776902444" Nov 02 15:10:41.016302 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:41.016264 5591 generic.go:356] "Generic (PLEG): container finished" podID="f945e8bb-6227-4304-a101-961d33363966" containerID="6896fa1157fdfe9714720f43bab4b12e9ff424234fcf42870b0e0f6c1463a00e" exitCode=0 Nov 02 15:10:41.016766 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:41.016320 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" event={"ID":"f945e8bb-6227-4304-a101-961d33363966","Type":"ContainerDied","Data":"6896fa1157fdfe9714720f43bab4b12e9ff424234fcf42870b0e0f6c1463a00e"} Nov 02 15:10:42.153722 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.153692 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:42.159761 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.159734 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f945e8bb-6227-4304-a101-961d33363966-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f945e8bb-6227-4304-a101-961d33363966-maas-api-xks-ca\") pod \"f945e8bb-6227-4304-a101-961d33363966\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " Nov 02 15:10:42.159895 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.159786 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f945e8bb-6227-4304-a101-961d33363966-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f945e8bb-6227-4304-a101-961d33363966-maas-api-openshift-ca\") pod \"f945e8bb-6227-4304-a101-961d33363966\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " Nov 02 15:10:42.159895 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.159844 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f945e8bb-6227-4304-a101-961d33363966-kube-api-access-w47ms\" (UniqueName: \"kubernetes.io/projected/f945e8bb-6227-4304-a101-961d33363966-kube-api-access-w47ms\") pod \"f945e8bb-6227-4304-a101-961d33363966\" (UID: \"f945e8bb-6227-4304-a101-961d33363966\") " Nov 02 15:10:42.160178 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.160155 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f945e8bb-6227-4304-a101-961d33363966-maas-api-openshift-ca" pod "f945e8bb-6227-4304-a101-961d33363966" (UID: "f945e8bb-6227-4304-a101-961d33363966"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:10:42.161808 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.161789 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f945e8bb-6227-4304-a101-961d33363966-kube-api-access-w47ms" pod "f945e8bb-6227-4304-a101-961d33363966" (UID: "f945e8bb-6227-4304-a101-961d33363966"). InnerVolumeSpecName "kube-api-access-w47ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:42.161895 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.161879 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f945e8bb-6227-4304-a101-961d33363966-maas-api-xks-ca" pod "f945e8bb-6227-4304-a101-961d33363966" (UID: "f945e8bb-6227-4304-a101-961d33363966"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:10:42.261524 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.261484 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f945e8bb-6227-4304-a101-961d33363966-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:42.261524 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.261524 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f945e8bb-6227-4304-a101-961d33363966-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:42.261524 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:42.261534 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w47ms\" (UniqueName: \"kubernetes.io/projected/f945e8bb-6227-4304-a101-961d33363966-kube-api-access-w47ms\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:43.026464 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:43.026429 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" Nov 02 15:10:43.026653 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:43.026461 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4" event={"ID":"f945e8bb-6227-4304-a101-961d33363966","Type":"ContainerDied","Data":"8b2e6a2ec22d802b23ec4bdaab66632d49b682ba1d722a0197269f2291f21307"} Nov 02 15:10:43.026653 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:43.026509 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b2e6a2ec22d802b23ec4bdaab66632d49b682ba1d722a0197269f2291f21307" Nov 02 15:10:48.265696 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:48.265648 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:49.761088 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.761043 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj"] Nov 02 15:10:49.761605 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.761426 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f945e8bb-6227-4304-a101-961d33363966" containerName="revoke-keys" Nov 02 15:10:49.761605 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.761439 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f945e8bb-6227-4304-a101-961d33363966" containerName="revoke-keys" Nov 02 15:10:49.761605 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.761529 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f945e8bb-6227-4304-a101-961d33363966" containerName="revoke-keys" Nov 02 15:10:49.764560 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.764539 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.782297 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.782270 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:10:49.782451 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.782367 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:10:49.794426 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.794297 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:10:49.794426 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.794389 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:10:49.796949 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.796921 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj"] Nov 02 15:10:49.813657 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.813624 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:10:49.827714 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.827676 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9hz\" (UniqueName: \"kubernetes.io/projected/39d1a927-1206-4b6b-852a-14bdd242e7c4-kube-api-access-gv9hz\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.827868 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.827815 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.827868 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.827854 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.928757 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.928724 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9hz\" (UniqueName: \"kubernetes.io/projected/39d1a927-1206-4b6b-852a-14bdd242e7c4-kube-api-access-gv9hz\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.928955 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.928894 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.928955 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.928943 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.929705 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.929679 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.931352 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.931330 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:49.951629 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:49.951595 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9hz\" (UniqueName: \"kubernetes.io/projected/39d1a927-1206-4b6b-852a-14bdd242e7c4-kube-api-access-gv9hz\") pod \"maas-api-revoke-sub-ec843018-cjmdj\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:50.078293 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:50.078204 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:50.378972 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:50.378945 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj"] Nov 02 15:10:50.381928 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:10:50.381898 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d1a927_1206_4b6b_852a_14bdd242e7c4.slice/crio-ba018a8a17305517f303db44c9366a7f199126b0f2696e131391fedfa242280b WatchSource:0}: Error finding container ba018a8a17305517f303db44c9366a7f199126b0f2696e131391fedfa242280b: Status 404 returned error can't find the container with id ba018a8a17305517f303db44c9366a7f199126b0f2696e131391fedfa242280b Nov 02 15:10:50.603718 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:50.603680 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:51.065268 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:51.065165 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" event={"ID":"39d1a927-1206-4b6b-852a-14bdd242e7c4","Type":"ContainerStarted","Data":"cfa99671758244b930b3bae03d20500d9369ab730f721afb59cc013b52ffe4d3"} Nov 02 15:10:51.065268 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:51.065221 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" event={"ID":"39d1a927-1206-4b6b-852a-14bdd242e7c4","Type":"ContainerStarted","Data":"ba018a8a17305517f303db44c9366a7f199126b0f2696e131391fedfa242280b"} Nov 02 15:10:51.106870 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:51.106812 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" podStartSLOduration=2.106799315 podStartE2EDuration="2.106799315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:51.10504375 +0000 UTC m=+5199.830665517" watchObservedRunningTime="2026-11-02 15:10:51.106799315 +0000 UTC m=+5199.832421079" Nov 02 15:10:51.554255 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:51.554213 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:52.071339 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:52.071301 5591 generic.go:356] "Generic (PLEG): container finished" podID="39d1a927-1206-4b6b-852a-14bdd242e7c4" containerID="cfa99671758244b930b3bae03d20500d9369ab730f721afb59cc013b52ffe4d3" exitCode=0 Nov 02 15:10:52.071793 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:52.071394 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" event={"ID":"39d1a927-1206-4b6b-852a-14bdd242e7c4","Type":"ContainerDied","Data":"cfa99671758244b930b3bae03d20500d9369ab730f721afb59cc013b52ffe4d3"} Nov 02 15:10:53.208491 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.208460 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:53.262559 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.262527 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d1a927-1206-4b6b-852a-14bdd242e7c4-kube-api-access-gv9hz\" (UniqueName: \"kubernetes.io/projected/39d1a927-1206-4b6b-852a-14bdd242e7c4-kube-api-access-gv9hz\") pod \"39d1a927-1206-4b6b-852a-14bdd242e7c4\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " Nov 02 15:10:53.262763 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.262593 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-openshift-ca\") pod \"39d1a927-1206-4b6b-852a-14bdd242e7c4\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " Nov 02 15:10:53.262763 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.262634 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-xks-ca\") pod \"39d1a927-1206-4b6b-852a-14bdd242e7c4\" (UID: \"39d1a927-1206-4b6b-852a-14bdd242e7c4\") " Nov 02 15:10:53.263029 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.262975 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-openshift-ca" pod "39d1a927-1206-4b6b-852a-14bdd242e7c4" (UID: "39d1a927-1206-4b6b-852a-14bdd242e7c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:10:53.264749 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.264726 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d1a927-1206-4b6b-852a-14bdd242e7c4-kube-api-access-gv9hz" pod "39d1a927-1206-4b6b-852a-14bdd242e7c4" (UID: "39d1a927-1206-4b6b-852a-14bdd242e7c4"). InnerVolumeSpecName "kube-api-access-gv9hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:53.264841 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.264767 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-xks-ca" pod "39d1a927-1206-4b6b-852a-14bdd242e7c4" (UID: "39d1a927-1206-4b6b-852a-14bdd242e7c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:10:53.363646 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.363560 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9hz\" (UniqueName: \"kubernetes.io/projected/39d1a927-1206-4b6b-852a-14bdd242e7c4-kube-api-access-gv9hz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:53.363646 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.363591 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:53.363646 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:53.363603 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d1a927-1206-4b6b-852a-14bdd242e7c4-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:10:54.081643 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:54.081610 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" Nov 02 15:10:54.081643 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:54.081631 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj" event={"ID":"39d1a927-1206-4b6b-852a-14bdd242e7c4","Type":"ContainerDied","Data":"ba018a8a17305517f303db44c9366a7f199126b0f2696e131391fedfa242280b"} Nov 02 15:10:54.081867 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:10:54.081677 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba018a8a17305517f303db44c9366a7f199126b0f2696e131391fedfa242280b" Nov 02 15:11:01.072734 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:11:01.072691 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:11:05.440272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:05.440235 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:11:17.292109 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.292069 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx"] Nov 02 15:11:17.292688 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.292665 5591 cpu_manager.go:414] "containerMap: removing container" podUID="39d1a927-1206-4b6b-852a-14bdd242e7c4" containerName="revoke-keys" Nov 02 15:11:17.292782 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.292690 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d1a927-1206-4b6b-852a-14bdd242e7c4" containerName="revoke-keys" Nov 02 15:11:17.292846 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.292836 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d1a927-1206-4b6b-852a-14bdd242e7c4" containerName="revoke-keys" Nov 02 15:11:17.296177 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.296155 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.315070 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.315003 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:11:17.325914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.325880 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:11:17.325914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.325887 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:11:17.326176 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.326053 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:11:17.328016 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.327965 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx"] Nov 02 15:11:17.395153 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.395119 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpmj\" (UniqueName: \"kubernetes.io/projected/7468325f-d3e1-48ef-9714-1060012b296b-kube-api-access-bzpmj\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.395329 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.395174 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.395329 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.395303 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.496402 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.496364 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpmj\" (UniqueName: \"kubernetes.io/projected/7468325f-d3e1-48ef-9714-1060012b296b-kube-api-access-bzpmj\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.496586 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.496417 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.496586 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.496468 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.497097 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.497070 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.498893 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.498860 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.519497 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.519474 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpmj\" (UniqueName: \"kubernetes.io/projected/7468325f-d3e1-48ef-9714-1060012b296b-kube-api-access-bzpmj\") pod \"maas-api-revoke-sub-046e1343-sdcrx\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.609618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.609542 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:17.906610 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:17.906582 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx"] Nov 02 15:11:17.909402 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:11:17.909371 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7468325f_d3e1_48ef_9714_1060012b296b.slice/crio-ab203f26f92c9da4da8b9f73d7034b70eabc23dcb2bbd98c1c8df30fd320aaa6 WatchSource:0}: Error finding container ab203f26f92c9da4da8b9f73d7034b70eabc23dcb2bbd98c1c8df30fd320aaa6: Status 404 returned error can't find the container with id ab203f26f92c9da4da8b9f73d7034b70eabc23dcb2bbd98c1c8df30fd320aaa6 Nov 02 15:11:18.232098 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:18.232009 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" event={"ID":"7468325f-d3e1-48ef-9714-1060012b296b","Type":"ContainerStarted","Data":"ab203f26f92c9da4da8b9f73d7034b70eabc23dcb2bbd98c1c8df30fd320aaa6"} Nov 02 15:11:19.239335 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:19.239293 5591 generic.go:356] "Generic (PLEG): container finished" podID="7468325f-d3e1-48ef-9714-1060012b296b" containerID="e6ab304c092163e6c95e2c565d3934c173a5f6eda7659ad93d86da61b7b7be30" exitCode=0 Nov 02 15:11:19.239816 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:19.239341 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" event={"ID":"7468325f-d3e1-48ef-9714-1060012b296b","Type":"ContainerDied","Data":"e6ab304c092163e6c95e2c565d3934c173a5f6eda7659ad93d86da61b7b7be30"} Nov 02 15:11:20.377566 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.377541 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:20.423828 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.423796 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-xks-ca\") pod \"7468325f-d3e1-48ef-9714-1060012b296b\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " Nov 02 15:11:20.424072 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.423871 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7468325f-d3e1-48ef-9714-1060012b296b-kube-api-access-bzpmj\" (UniqueName: \"kubernetes.io/projected/7468325f-d3e1-48ef-9714-1060012b296b-kube-api-access-bzpmj\") pod \"7468325f-d3e1-48ef-9714-1060012b296b\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " Nov 02 15:11:20.424072 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.423930 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-openshift-ca\") pod \"7468325f-d3e1-48ef-9714-1060012b296b\" (UID: \"7468325f-d3e1-48ef-9714-1060012b296b\") " Nov 02 15:11:20.424620 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.424589 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-openshift-ca" pod "7468325f-d3e1-48ef-9714-1060012b296b" (UID: "7468325f-d3e1-48ef-9714-1060012b296b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:11:20.426573 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.426543 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-xks-ca" pod "7468325f-d3e1-48ef-9714-1060012b296b" (UID: "7468325f-d3e1-48ef-9714-1060012b296b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:11:20.427627 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.427598 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7468325f-d3e1-48ef-9714-1060012b296b-kube-api-access-bzpmj" pod "7468325f-d3e1-48ef-9714-1060012b296b" (UID: "7468325f-d3e1-48ef-9714-1060012b296b"). InnerVolumeSpecName "kube-api-access-bzpmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:20.524835 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.524744 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:20.524835 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.524776 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzpmj\" (UniqueName: \"kubernetes.io/projected/7468325f-d3e1-48ef-9714-1060012b296b-kube-api-access-bzpmj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:20.524835 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:20.524788 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7468325f-d3e1-48ef-9714-1060012b296b-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:21.250185 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:21.250150 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" Nov 02 15:11:21.250375 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:21.250185 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx" event={"ID":"7468325f-d3e1-48ef-9714-1060012b296b","Type":"ContainerDied","Data":"ab203f26f92c9da4da8b9f73d7034b70eabc23dcb2bbd98c1c8df30fd320aaa6"} Nov 02 15:11:21.250375 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:21.250237 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab203f26f92c9da4da8b9f73d7034b70eabc23dcb2bbd98c1c8df30fd320aaa6" Nov 02 15:11:27.028162 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.028080 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l"] Nov 02 15:11:27.039870 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.039843 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fv6l"] Nov 02 15:11:27.429467 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.429430 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055daa10-1749-4b7f-97b4-6cb59eee93dc" path="/var/lib/kubelet/pods/055daa10-1749-4b7f-97b4-6cb59eee93dc/volumes" Nov 02 15:11:27.609759 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.609723 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz"] Nov 02 15:11:27.610160 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.610146 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7468325f-d3e1-48ef-9714-1060012b296b" containerName="revoke-keys" Nov 02 15:11:27.610218 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.610162 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7468325f-d3e1-48ef-9714-1060012b296b" containerName="revoke-keys" Nov 02 15:11:27.610261 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.610241 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7468325f-d3e1-48ef-9714-1060012b296b" containerName="revoke-keys" Nov 02 15:11:27.613377 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.613361 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.630837 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.630809 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:11:27.631127 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.631106 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:11:27.631245 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.631222 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:11:27.633769 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.633746 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz"] Nov 02 15:11:27.690013 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.689911 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.690013 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.689952 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbvr\" (UniqueName: \"kubernetes.io/projected/27e360a8-a775-468c-8603-2a8435a765d1-kube-api-access-5cbvr\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.690224 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.690060 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.729376 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.729345 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:11:27.790692 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.790648 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.790896 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.790743 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.790896 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.790777 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbvr\" (UniqueName: \"kubernetes.io/projected/27e360a8-a775-468c-8603-2a8435a765d1-kube-api-access-5cbvr\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.791390 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.791355 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.793112 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.793092 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.814235 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.814211 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbvr\" (UniqueName: \"kubernetes.io/projected/27e360a8-a775-468c-8603-2a8435a765d1-kube-api-access-5cbvr\") pod \"maas-api-revoke-sub-d3fba056-6snqz\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:27.926154 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:27.926119 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:28.223526 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:28.223482 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz"] Nov 02 15:11:28.226632 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:11:28.226603 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e360a8_a775_468c_8603_2a8435a765d1.slice/crio-caf229a61dc3137d48702643006708f5301892dc1f04b989acd8fd5fdf88f90e WatchSource:0}: Error finding container caf229a61dc3137d48702643006708f5301892dc1f04b989acd8fd5fdf88f90e: Status 404 returned error can't find the container with id caf229a61dc3137d48702643006708f5301892dc1f04b989acd8fd5fdf88f90e Nov 02 15:11:28.284114 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:28.284078 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" event={"ID":"27e360a8-a775-468c-8603-2a8435a765d1","Type":"ContainerStarted","Data":"caf229a61dc3137d48702643006708f5301892dc1f04b989acd8fd5fdf88f90e"} Nov 02 15:11:28.706889 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:28.706850 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:11:29.290949 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:29.290838 5591 generic.go:356] "Generic (PLEG): container finished" podID="27e360a8-a775-468c-8603-2a8435a765d1" containerID="53fb2c8c2ba2d6296eab6433ed3c4b76e70a65aa8167d01726a9fd93e2ca9c32" exitCode=0 Nov 02 15:11:29.290949 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:29.290909 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" event={"ID":"27e360a8-a775-468c-8603-2a8435a765d1","Type":"ContainerDied","Data":"53fb2c8c2ba2d6296eab6433ed3c4b76e70a65aa8167d01726a9fd93e2ca9c32"} Nov 02 15:11:30.636285 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.636252 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:30.710689 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.710609 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:30.723383 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.723346 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e360a8-a775-468c-8603-2a8435a765d1-kube-api-access-5cbvr\" (UniqueName: \"kubernetes.io/projected/27e360a8-a775-468c-8603-2a8435a765d1-kube-api-access-5cbvr\") pod \"27e360a8-a775-468c-8603-2a8435a765d1\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " Nov 02 15:11:30.723546 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.723454 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-openshift-ca\") pod \"27e360a8-a775-468c-8603-2a8435a765d1\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " Nov 02 15:11:30.723546 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.723528 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-xks-ca\") pod \"27e360a8-a775-468c-8603-2a8435a765d1\" (UID: \"27e360a8-a775-468c-8603-2a8435a765d1\") " Nov 02 15:11:30.723884 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.723855 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-openshift-ca" pod "27e360a8-a775-468c-8603-2a8435a765d1" (UID: "27e360a8-a775-468c-8603-2a8435a765d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:11:30.725483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.725456 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e360a8-a775-468c-8603-2a8435a765d1-kube-api-access-5cbvr" pod "27e360a8-a775-468c-8603-2a8435a765d1" (UID: "27e360a8-a775-468c-8603-2a8435a765d1"). InnerVolumeSpecName "kube-api-access-5cbvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:30.725589 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.725564 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-xks-ca" pod "27e360a8-a775-468c-8603-2a8435a765d1" (UID: "27e360a8-a775-468c-8603-2a8435a765d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:11:30.824748 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.824711 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:30.824748 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.824745 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e360a8-a775-468c-8603-2a8435a765d1-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:30.824748 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:30.824755 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbvr\" (UniqueName: \"kubernetes.io/projected/27e360a8-a775-468c-8603-2a8435a765d1-kube-api-access-5cbvr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:31.121484 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:11:31.121444 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:11:31.308647 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:31.308594 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" event={"ID":"27e360a8-a775-468c-8603-2a8435a765d1","Type":"ContainerDied","Data":"caf229a61dc3137d48702643006708f5301892dc1f04b989acd8fd5fdf88f90e"} Nov 02 15:11:31.308966 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:31.308931 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caf229a61dc3137d48702643006708f5301892dc1f04b989acd8fd5fdf88f90e" Nov 02 15:11:31.309167 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:31.308628 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz" Nov 02 15:11:38.940093 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:38.940051 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:11:43.638898 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:43.638853 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:11:44.032598 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:44.032506 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w"] Nov 02 15:11:44.046538 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:44.046483 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4lj4w"] Nov 02 15:11:45.428417 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:45.428378 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70bf106-46ab-44c5-94e1-4c1ccc807f9e" path="/var/lib/kubelet/pods/b70bf106-46ab-44c5-94e1-4c1ccc807f9e/volumes" Nov 02 15:11:52.130507 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.130468 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b"] Nov 02 15:11:52.130906 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.130869 5591 cpu_manager.go:414] "containerMap: removing container" podUID="27e360a8-a775-468c-8603-2a8435a765d1" containerName="revoke-keys" Nov 02 15:11:52.130906 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.130881 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e360a8-a775-468c-8603-2a8435a765d1" containerName="revoke-keys" Nov 02 15:11:52.131010 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.130945 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e360a8-a775-468c-8603-2a8435a765d1" containerName="revoke-keys" Nov 02 15:11:52.134994 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.134969 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.154169 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.154137 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b"] Nov 02 15:11:52.163948 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.163909 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:11:52.164088 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.163908 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:11:52.164088 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.163945 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:11:52.164088 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.163995 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:11:52.256622 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.256589 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.256797 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.256751 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.256843 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.256803 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkg7\" (UniqueName: \"kubernetes.io/projected/183126d4-3d0a-48af-87fb-cc0c18d6d076-kube-api-access-jvkg7\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.358181 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.358144 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.358181 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.358182 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkg7\" (UniqueName: \"kubernetes.io/projected/183126d4-3d0a-48af-87fb-cc0c18d6d076-kube-api-access-jvkg7\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.358422 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.358202 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.358894 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.358867 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.360561 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.360542 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.381395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.381364 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkg7\" (UniqueName: \"kubernetes.io/projected/183126d4-3d0a-48af-87fb-cc0c18d6d076-kube-api-access-jvkg7\") pod \"maas-api-revoke-sub-ae594979-mc46b\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.447796 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.447768 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:52.747661 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:52.747575 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b"] Nov 02 15:11:52.750828 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:11:52.750791 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183126d4_3d0a_48af_87fb_cc0c18d6d076.slice/crio-14d4074419752f211039a01eb978f1b77d97264e727b94228b04b445890cadbc WatchSource:0}: Error finding container 14d4074419752f211039a01eb978f1b77d97264e727b94228b04b445890cadbc: Status 404 returned error can't find the container with id 14d4074419752f211039a01eb978f1b77d97264e727b94228b04b445890cadbc Nov 02 15:11:53.433923 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:53.433889 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" event={"ID":"183126d4-3d0a-48af-87fb-cc0c18d6d076","Type":"ContainerStarted","Data":"d04f272202db23395acf62bd3a4472f0bec83ff2aa95e693d9ba75d736752ce7"} Nov 02 15:11:53.434345 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:53.433938 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" event={"ID":"183126d4-3d0a-48af-87fb-cc0c18d6d076","Type":"ContainerStarted","Data":"14d4074419752f211039a01eb978f1b77d97264e727b94228b04b445890cadbc"} Nov 02 15:11:53.474244 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:53.474197 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" podStartSLOduration=1.474183854 podStartE2EDuration="1.474183854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:53.473458033 +0000 UTC m=+5262.199079803" watchObservedRunningTime="2026-11-02 15:11:53.474183854 +0000 UTC m=+5262.199805621" Nov 02 15:11:54.440072 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:54.440031 5591 generic.go:356] "Generic (PLEG): container finished" podID="183126d4-3d0a-48af-87fb-cc0c18d6d076" containerID="d04f272202db23395acf62bd3a4472f0bec83ff2aa95e693d9ba75d736752ce7" exitCode=0 Nov 02 15:11:54.440476 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:54.440108 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" event={"ID":"183126d4-3d0a-48af-87fb-cc0c18d6d076","Type":"ContainerDied","Data":"d04f272202db23395acf62bd3a4472f0bec83ff2aa95e693d9ba75d736752ce7"} Nov 02 15:11:55.577542 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.577516 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:55.587530 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.587507 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-openshift-ca\") pod \"183126d4-3d0a-48af-87fb-cc0c18d6d076\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " Nov 02 15:11:55.587660 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.587543 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183126d4-3d0a-48af-87fb-cc0c18d6d076-kube-api-access-jvkg7\" (UniqueName: \"kubernetes.io/projected/183126d4-3d0a-48af-87fb-cc0c18d6d076-kube-api-access-jvkg7\") pod \"183126d4-3d0a-48af-87fb-cc0c18d6d076\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " Nov 02 15:11:55.587660 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.587592 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-xks-ca\") pod \"183126d4-3d0a-48af-87fb-cc0c18d6d076\" (UID: \"183126d4-3d0a-48af-87fb-cc0c18d6d076\") " Nov 02 15:11:55.587897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.587873 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-openshift-ca" pod "183126d4-3d0a-48af-87fb-cc0c18d6d076" (UID: "183126d4-3d0a-48af-87fb-cc0c18d6d076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:11:55.589536 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.589513 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183126d4-3d0a-48af-87fb-cc0c18d6d076-kube-api-access-jvkg7" pod "183126d4-3d0a-48af-87fb-cc0c18d6d076" (UID: "183126d4-3d0a-48af-87fb-cc0c18d6d076"). InnerVolumeSpecName "kube-api-access-jvkg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:11:55.589618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.589569 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-xks-ca" pod "183126d4-3d0a-48af-87fb-cc0c18d6d076" (UID: "183126d4-3d0a-48af-87fb-cc0c18d6d076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:11:55.688380 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.688346 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:55.688380 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.688375 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkg7\" (UniqueName: \"kubernetes.io/projected/183126d4-3d0a-48af-87fb-cc0c18d6d076-kube-api-access-jvkg7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:55.688380 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:55.688386 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183126d4-3d0a-48af-87fb-cc0c18d6d076-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:11:56.450568 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:56.450531 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" Nov 02 15:11:56.450765 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:56.450569 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b" event={"ID":"183126d4-3d0a-48af-87fb-cc0c18d6d076","Type":"ContainerDied","Data":"14d4074419752f211039a01eb978f1b77d97264e727b94228b04b445890cadbc"} Nov 02 15:11:56.450765 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:11:56.450629 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d4074419752f211039a01eb978f1b77d97264e727b94228b04b445890cadbc" Nov 02 15:12:01.072783 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:12:01.072743 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:12:03.509575 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:03.509531 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:12:07.349009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.345904 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr"] Nov 02 15:12:07.364595 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.364560 5591 cpu_manager.go:414] "containerMap: removing container" podUID="183126d4-3d0a-48af-87fb-cc0c18d6d076" containerName="revoke-keys" Nov 02 15:12:07.364833 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.364811 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183126d4-3d0a-48af-87fb-cc0c18d6d076" containerName="revoke-keys" Nov 02 15:12:07.365801 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.365775 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="183126d4-3d0a-48af-87fb-cc0c18d6d076" containerName="revoke-keys" Nov 02 15:12:07.509262 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.509224 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr"] Nov 02 15:12:07.517441 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.517408 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.554330 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.554297 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:12:07.554827 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.554805 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:12:07.559073 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.556310 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:12:07.559252 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.556478 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:12:07.613695 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.613602 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnz6l\" (UniqueName: \"kubernetes.io/projected/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-kube-api-access-mnz6l\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.614064 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.614029 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.614202 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.614097 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.715437 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.715396 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnz6l\" (UniqueName: \"kubernetes.io/projected/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-kube-api-access-mnz6l\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.715615 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.715467 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.715615 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.715512 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.716316 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.716287 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.717775 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.717750 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.760900 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.760858 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnz6l\" (UniqueName: \"kubernetes.io/projected/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-kube-api-access-mnz6l\") pod \"maas-api-revoke-sub-a4aef7d7-497nr\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:07.869508 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:07.869137 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:08.268165 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:08.268052 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr"] Nov 02 15:12:08.524226 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:08.524070 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" event={"ID":"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d","Type":"ContainerStarted","Data":"892e25263f271778f7bc46dd7251790c8cd496d5559d7d254bb8c9fa56776564"} Nov 02 15:12:09.157102 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:09.157060 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:12:09.530293 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:09.530205 5591 generic.go:356] "Generic (PLEG): container finished" podID="7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d" containerID="23fa76c789c432e37bab2923c8eca3b94a07cde6cba8dbebd0082c46bd4a8503" exitCode=0 Nov 02 15:12:09.530693 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:09.530294 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" event={"ID":"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d","Type":"ContainerDied","Data":"23fa76c789c432e37bab2923c8eca3b94a07cde6cba8dbebd0082c46bd4a8503"} Nov 02 15:12:10.664542 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.664518 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:10.741916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.741867 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-xks-ca\") pod \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " Nov 02 15:12:10.742124 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.741923 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-openshift-ca\") pod \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " Nov 02 15:12:10.742124 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.741961 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-kube-api-access-mnz6l\" (UniqueName: \"kubernetes.io/projected/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-kube-api-access-mnz6l\") pod \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\" (UID: \"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d\") " Nov 02 15:12:10.742320 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.742298 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-openshift-ca" pod "7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d" (UID: "7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:12:10.744039 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.744014 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-xks-ca" pod "7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d" (UID: "7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:12:10.744188 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.744109 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-kube-api-access-mnz6l" pod "7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d" (UID: "7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d"). InnerVolumeSpecName "kube-api-access-mnz6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:12:10.842916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.842824 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:10.842916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.842859 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:10.842916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:10.842872 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnz6l\" (UniqueName: \"kubernetes.io/projected/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d-kube-api-access-mnz6l\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:11.541258 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:11.541219 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" event={"ID":"7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d","Type":"ContainerDied","Data":"892e25263f271778f7bc46dd7251790c8cd496d5559d7d254bb8c9fa56776564"} Nov 02 15:12:11.541443 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:11.541274 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="892e25263f271778f7bc46dd7251790c8cd496d5559d7d254bb8c9fa56776564" Nov 02 15:12:11.541443 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:11.541230 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr" Nov 02 15:12:13.531631 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:13.531587 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:12:14.029679 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:14.029639 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql"] Nov 02 15:12:14.041776 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:14.041743 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3929db6-pxqql"] Nov 02 15:12:15.429445 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:15.429404 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e879ae80-6cb5-4fb4-97b4-21141ca898ac" path="/var/lib/kubelet/pods/e879ae80-6cb5-4fb4-97b4-21141ca898ac/volumes" Nov 02 15:12:23.033109 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:23.033063 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:12:24.326569 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.326528 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k"] Nov 02 15:12:24.327273 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.327252 5591 cpu_manager.go:414] "containerMap: removing container" podUID="7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d" containerName="revoke-keys" Nov 02 15:12:24.327390 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.327275 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d" containerName="revoke-keys" Nov 02 15:12:24.327390 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.327371 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d" containerName="revoke-keys" Nov 02 15:12:24.332335 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.332309 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.357814 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.357777 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k"] Nov 02 15:12:24.358942 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.358917 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:12:24.359255 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.359237 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:12:24.359475 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.359457 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:12:24.359654 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.359638 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:12:24.475499 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.475457 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.475848 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.475823 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wk4q\" (UniqueName: \"kubernetes.io/projected/ed0a10fe-a04e-4b42-b3ad-160884777826-kube-api-access-7wk4q\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.476082 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.476063 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.577556 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.577450 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.578478 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.578449 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wk4q\" (UniqueName: \"kubernetes.io/projected/ed0a10fe-a04e-4b42-b3ad-160884777826-kube-api-access-7wk4q\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.578702 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.578683 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.579655 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.579630 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.601977 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.601947 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.612522 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.612490 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wk4q\" (UniqueName: \"kubernetes.io/projected/ed0a10fe-a04e-4b42-b3ad-160884777826-kube-api-access-7wk4q\") pod \"maas-api-revoke-sub-79805ca9-mnk4k\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:24.671008 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:24.670902 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:25.097551 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:25.097516 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k"] Nov 02 15:12:25.105006 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:12:25.104957 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0a10fe_a04e_4b42_b3ad_160884777826.slice/crio-62801e857931f1ee77d4dbb4a6779b9d9f9afdc74171ebc7171521b1df4a0e79 WatchSource:0}: Error finding container 62801e857931f1ee77d4dbb4a6779b9d9f9afdc74171ebc7171521b1df4a0e79: Status 404 returned error can't find the container with id 62801e857931f1ee77d4dbb4a6779b9d9f9afdc74171ebc7171521b1df4a0e79 Nov 02 15:12:25.624727 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:25.624645 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" event={"ID":"ed0a10fe-a04e-4b42-b3ad-160884777826","Type":"ContainerStarted","Data":"e4a4fbbbac1fd492b7416a33a6d862331e8ff5493128f07e14f96d40d95ac59a"} Nov 02 15:12:25.624727 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:25.624707 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" event={"ID":"ed0a10fe-a04e-4b42-b3ad-160884777826","Type":"ContainerStarted","Data":"62801e857931f1ee77d4dbb4a6779b9d9f9afdc74171ebc7171521b1df4a0e79"} Nov 02 15:12:25.672007 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:25.671692 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" podStartSLOduration=1.671672392 podStartE2EDuration="1.671672392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:25.670360202 +0000 UTC m=+5294.395981973" watchObservedRunningTime="2026-11-02 15:12:25.671672392 +0000 UTC m=+5294.397294165" Nov 02 15:12:26.630633 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:26.630542 5591 generic.go:356] "Generic (PLEG): container finished" podID="ed0a10fe-a04e-4b42-b3ad-160884777826" containerID="e4a4fbbbac1fd492b7416a33a6d862331e8ff5493128f07e14f96d40d95ac59a" exitCode=0 Nov 02 15:12:26.631039 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:26.630624 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" event={"ID":"ed0a10fe-a04e-4b42-b3ad-160884777826","Type":"ContainerDied","Data":"e4a4fbbbac1fd492b7416a33a6d862331e8ff5493128f07e14f96d40d95ac59a"} Nov 02 15:12:27.765218 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.765189 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:27.834378 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.834338 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-openshift-ca\") pod \"ed0a10fe-a04e-4b42-b3ad-160884777826\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " Nov 02 15:12:27.834575 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.834436 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0a10fe-a04e-4b42-b3ad-160884777826-kube-api-access-7wk4q\" (UniqueName: \"kubernetes.io/projected/ed0a10fe-a04e-4b42-b3ad-160884777826-kube-api-access-7wk4q\") pod \"ed0a10fe-a04e-4b42-b3ad-160884777826\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " Nov 02 15:12:27.834575 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.834466 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-xks-ca\") pod \"ed0a10fe-a04e-4b42-b3ad-160884777826\" (UID: \"ed0a10fe-a04e-4b42-b3ad-160884777826\") " Nov 02 15:12:27.834741 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.834714 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-openshift-ca" pod "ed0a10fe-a04e-4b42-b3ad-160884777826" (UID: "ed0a10fe-a04e-4b42-b3ad-160884777826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:12:27.834842 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.834824 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:27.836658 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.836630 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-xks-ca" pod "ed0a10fe-a04e-4b42-b3ad-160884777826" (UID: "ed0a10fe-a04e-4b42-b3ad-160884777826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:12:27.836774 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.836701 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0a10fe-a04e-4b42-b3ad-160884777826-kube-api-access-7wk4q" pod "ed0a10fe-a04e-4b42-b3ad-160884777826" (UID: "ed0a10fe-a04e-4b42-b3ad-160884777826"). InnerVolumeSpecName "kube-api-access-7wk4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:12:27.936077 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.935965 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wk4q\" (UniqueName: \"kubernetes.io/projected/ed0a10fe-a04e-4b42-b3ad-160884777826-kube-api-access-7wk4q\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:27.936077 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:27.936025 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0a10fe-a04e-4b42-b3ad-160884777826-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:28.640798 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:28.640763 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" Nov 02 15:12:28.641034 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:28.640804 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k" event={"ID":"ed0a10fe-a04e-4b42-b3ad-160884777826","Type":"ContainerDied","Data":"62801e857931f1ee77d4dbb4a6779b9d9f9afdc74171ebc7171521b1df4a0e79"} Nov 02 15:12:28.641034 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:28.640853 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62801e857931f1ee77d4dbb4a6779b9d9f9afdc74171ebc7171521b1df4a0e79" Nov 02 15:12:28.648231 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:28.648203 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:12:30.838283 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:30.838236 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:31.071962 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:12:31.071927 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:12:36.649031 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:36.648991 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:12:38.102006 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.101816 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh"] Nov 02 15:12:38.102543 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.102413 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ed0a10fe-a04e-4b42-b3ad-160884777826" containerName="revoke-keys" Nov 02 15:12:38.102543 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.102436 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0a10fe-a04e-4b42-b3ad-160884777826" containerName="revoke-keys" Nov 02 15:12:38.102672 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.102571 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0a10fe-a04e-4b42-b3ad-160884777826" containerName="revoke-keys" Nov 02 15:12:38.110036 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.107243 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.125518 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.125486 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh"] Nov 02 15:12:38.150009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.139949 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:12:38.150009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.146362 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:12:38.163005 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.160504 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:12:38.163005 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.160617 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:12:38.258011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.254497 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fcwz\" (UniqueName: \"kubernetes.io/projected/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-kube-api-access-2fcwz\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.258011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.254575 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.258011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.254676 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.363059 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.362471 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.363059 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.362580 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fcwz\" (UniqueName: \"kubernetes.io/projected/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-kube-api-access-2fcwz\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.363059 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.362625 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.369009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.363565 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.410863 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.410828 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.422022 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.421918 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fcwz\" (UniqueName: \"kubernetes.io/projected/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-kube-api-access-2fcwz\") pod \"maas-api-revoke-sub-0b7e8edb-czpwh\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.424628 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.424602 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:38.864272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:38.864230 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh"] Nov 02 15:12:38.884218 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:12:38.884145 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5a23c5_6c64_4ece_bb96_7e461766ebdf.slice/crio-99cf84256112e7357ca63af9fce2b1ce4f0d4c0393a0a52a46b3757a5877652a WatchSource:0}: Error finding container 99cf84256112e7357ca63af9fce2b1ce4f0d4c0393a0a52a46b3757a5877652a: Status 404 returned error can't find the container with id 99cf84256112e7357ca63af9fce2b1ce4f0d4c0393a0a52a46b3757a5877652a Nov 02 15:12:39.717677 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:39.717620 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" event={"ID":"ca5a23c5-6c64-4ece-bb96-7e461766ebdf","Type":"ContainerStarted","Data":"234af82e33810b558c5c27505842bcb2eecb4fa26ccb7a4a9cbbf96eae8acc7c"} Nov 02 15:12:39.718262 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:39.717698 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" event={"ID":"ca5a23c5-6c64-4ece-bb96-7e461766ebdf","Type":"ContainerStarted","Data":"99cf84256112e7357ca63af9fce2b1ce4f0d4c0393a0a52a46b3757a5877652a"} Nov 02 15:12:39.759552 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:39.759482 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" podStartSLOduration=1.759462805 podStartE2EDuration="1.759462805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:39.757955054 +0000 UTC m=+5308.483576827" watchObservedRunningTime="2026-11-02 15:12:39.759462805 +0000 UTC m=+5308.485084576" Nov 02 15:12:40.723916 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:40.723882 5591 generic.go:356] "Generic (PLEG): container finished" podID="ca5a23c5-6c64-4ece-bb96-7e461766ebdf" containerID="234af82e33810b558c5c27505842bcb2eecb4fa26ccb7a4a9cbbf96eae8acc7c" exitCode=0 Nov 02 15:12:40.724359 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:40.723974 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" event={"ID":"ca5a23c5-6c64-4ece-bb96-7e461766ebdf","Type":"ContainerDied","Data":"234af82e33810b558c5c27505842bcb2eecb4fa26ccb7a4a9cbbf96eae8acc7c"} Nov 02 15:12:41.857237 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:41.857210 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:42.027148 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.027059 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-openshift-ca\") pod \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " Nov 02 15:12:42.027148 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.027118 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-xks-ca\") pod \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " Nov 02 15:12:42.027419 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.027221 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-kube-api-access-2fcwz\" (UniqueName: \"kubernetes.io/projected/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-kube-api-access-2fcwz\") pod \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\" (UID: \"ca5a23c5-6c64-4ece-bb96-7e461766ebdf\") " Nov 02 15:12:42.027535 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.027513 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-openshift-ca" pod "ca5a23c5-6c64-4ece-bb96-7e461766ebdf" (UID: "ca5a23c5-6c64-4ece-bb96-7e461766ebdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:12:42.029272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.029251 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-xks-ca" pod "ca5a23c5-6c64-4ece-bb96-7e461766ebdf" (UID: "ca5a23c5-6c64-4ece-bb96-7e461766ebdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:12:42.029352 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.029323 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-kube-api-access-2fcwz" pod "ca5a23c5-6c64-4ece-bb96-7e461766ebdf" (UID: "ca5a23c5-6c64-4ece-bb96-7e461766ebdf"). InnerVolumeSpecName "kube-api-access-2fcwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:12:42.128897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.128857 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fcwz\" (UniqueName: \"kubernetes.io/projected/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-kube-api-access-2fcwz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:42.128897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.128893 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:42.128897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.128903 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5a23c5-6c64-4ece-bb96-7e461766ebdf-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:42.248483 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.248447 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:12:42.734339 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.734307 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" Nov 02 15:12:42.734526 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.734308 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh" event={"ID":"ca5a23c5-6c64-4ece-bb96-7e461766ebdf","Type":"ContainerDied","Data":"99cf84256112e7357ca63af9fce2b1ce4f0d4c0393a0a52a46b3757a5877652a"} Nov 02 15:12:42.734526 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:42.734429 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99cf84256112e7357ca63af9fce2b1ce4f0d4c0393a0a52a46b3757a5877652a" Nov 02 15:12:48.819254 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.819215 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp"] Nov 02 15:12:48.819675 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.819660 5591 cpu_manager.go:414] "containerMap: removing container" podUID="ca5a23c5-6c64-4ece-bb96-7e461766ebdf" containerName="revoke-keys" Nov 02 15:12:48.819744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.819676 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5a23c5-6c64-4ece-bb96-7e461766ebdf" containerName="revoke-keys" Nov 02 15:12:48.819795 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.819747 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5a23c5-6c64-4ece-bb96-7e461766ebdf" containerName="revoke-keys" Nov 02 15:12:48.823132 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.823108 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:48.852200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.852138 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:12:48.852393 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.852285 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:12:48.852393 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.852308 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:12:48.852694 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.852667 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:12:48.854328 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.854308 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp"] Nov 02 15:12:48.892705 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.892671 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df96964-40b2-4547-9328-850166719c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:48.892881 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.892730 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df96964-40b2-4547-9328-850166719c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:48.892881 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.892858 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rmsp\" (UniqueName: \"kubernetes.io/projected/5df96964-40b2-4547-9328-850166719c72-kube-api-access-5rmsp\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:48.993441 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.993409 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df96964-40b2-4547-9328-850166719c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:48.993629 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.993453 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df96964-40b2-4547-9328-850166719c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:48.993629 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.993487 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rmsp\" (UniqueName: \"kubernetes.io/projected/5df96964-40b2-4547-9328-850166719c72-kube-api-access-5rmsp\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:48.994101 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.994074 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df96964-40b2-4547-9328-850166719c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:48.995857 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:48.995826 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df96964-40b2-4547-9328-850166719c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:49.017107 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:49.017080 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rmsp\" (UniqueName: \"kubernetes.io/projected/5df96964-40b2-4547-9328-850166719c72-kube-api-access-5rmsp\") pod \"maas-api-revoke-sub-3d4156f1-qrqdp\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:49.136689 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:49.136662 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:49.433876 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:49.433847 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp"] Nov 02 15:12:49.436931 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:12:49.436901 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df96964_40b2_4547_9328_850166719c72.slice/crio-4dc0fddc994855565e1ad4457b2f8f2e02de35855397b1d1430137a087b95818 WatchSource:0}: Error finding container 4dc0fddc994855565e1ad4457b2f8f2e02de35855397b1d1430137a087b95818: Status 404 returned error can't find the container with id 4dc0fddc994855565e1ad4457b2f8f2e02de35855397b1d1430137a087b95818 Nov 02 15:12:49.770008 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:49.769963 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" event={"ID":"5df96964-40b2-4547-9328-850166719c72","Type":"ContainerStarted","Data":"3b3217a7ef01cbb58ac64693b3839845b62bac6fc2a6b5c0b1eef6b0403389ae"} Nov 02 15:12:49.770145 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:49.770024 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" event={"ID":"5df96964-40b2-4547-9328-850166719c72","Type":"ContainerStarted","Data":"4dc0fddc994855565e1ad4457b2f8f2e02de35855397b1d1430137a087b95818"} Nov 02 15:12:49.809675 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:49.809621 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" podStartSLOduration=1.809605038 podStartE2EDuration="1.809605038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:49.8093025 +0000 UTC m=+5318.534924269" watchObservedRunningTime="2026-11-02 15:12:49.809605038 +0000 UTC m=+5318.535226807" Nov 02 15:12:50.781898 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:50.781852 5591 generic.go:356] "Generic (PLEG): container finished" podID="5df96964-40b2-4547-9328-850166719c72" containerID="3b3217a7ef01cbb58ac64693b3839845b62bac6fc2a6b5c0b1eef6b0403389ae" exitCode=0 Nov 02 15:12:50.782386 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:50.782026 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" event={"ID":"5df96964-40b2-4547-9328-850166719c72","Type":"ContainerDied","Data":"3b3217a7ef01cbb58ac64693b3839845b62bac6fc2a6b5c0b1eef6b0403389ae"} Nov 02 15:12:51.913458 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:51.913429 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:52.019941 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.019896 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df96964-40b2-4547-9328-850166719c72-kube-api-access-5rmsp\" (UniqueName: \"kubernetes.io/projected/5df96964-40b2-4547-9328-850166719c72-kube-api-access-5rmsp\") pod \"5df96964-40b2-4547-9328-850166719c72\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " Nov 02 15:12:52.020124 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.019968 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df96964-40b2-4547-9328-850166719c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df96964-40b2-4547-9328-850166719c72-maas-api-openshift-ca\") pod \"5df96964-40b2-4547-9328-850166719c72\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " Nov 02 15:12:52.020124 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.020028 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df96964-40b2-4547-9328-850166719c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df96964-40b2-4547-9328-850166719c72-maas-api-xks-ca\") pod \"5df96964-40b2-4547-9328-850166719c72\" (UID: \"5df96964-40b2-4547-9328-850166719c72\") " Nov 02 15:12:52.020424 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.020395 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df96964-40b2-4547-9328-850166719c72-maas-api-openshift-ca" pod "5df96964-40b2-4547-9328-850166719c72" (UID: "5df96964-40b2-4547-9328-850166719c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:12:52.022078 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.022050 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df96964-40b2-4547-9328-850166719c72-maas-api-xks-ca" pod "5df96964-40b2-4547-9328-850166719c72" (UID: "5df96964-40b2-4547-9328-850166719c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:12:52.022209 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.022175 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df96964-40b2-4547-9328-850166719c72-kube-api-access-5rmsp" pod "5df96964-40b2-4547-9328-850166719c72" (UID: "5df96964-40b2-4547-9328-850166719c72"). InnerVolumeSpecName "kube-api-access-5rmsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:12:52.120700 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.120611 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rmsp\" (UniqueName: \"kubernetes.io/projected/5df96964-40b2-4547-9328-850166719c72-kube-api-access-5rmsp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:52.120700 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.120644 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df96964-40b2-4547-9328-850166719c72-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:52.120700 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.120654 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df96964-40b2-4547-9328-850166719c72-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:12:52.792530 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.792454 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" Nov 02 15:12:52.792530 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.792481 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp" event={"ID":"5df96964-40b2-4547-9328-850166719c72","Type":"ContainerDied","Data":"4dc0fddc994855565e1ad4457b2f8f2e02de35855397b1d1430137a087b95818"} Nov 02 15:12:52.792530 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:12:52.792524 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc0fddc994855565e1ad4457b2f8f2e02de35855397b1d1430137a087b95818" Nov 02 15:13:00.155917 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:00.155872 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:13:01.075934 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:13:01.075867 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:13:05.842009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:05.840081 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:13:05.842009 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:05.840149 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:13:06.828170 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:06.828091 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:13:06.981746 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:06.981708 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp"] Nov 02 15:13:06.993588 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:06.993549 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrqdp"] Nov 02 15:13:07.023214 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.023176 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm"] Nov 02 15:13:07.023604 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.023587 5591 cpu_manager.go:414] "containerMap: removing container" podUID="5df96964-40b2-4547-9328-850166719c72" containerName="revoke-keys" Nov 02 15:13:07.023604 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.023603 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df96964-40b2-4547-9328-850166719c72" containerName="revoke-keys" Nov 02 15:13:07.023757 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.023661 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df96964-40b2-4547-9328-850166719c72" containerName="revoke-keys" Nov 02 15:13:07.026914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.026893 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.044245 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.044217 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:13:07.044601 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.044582 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:13:07.044763 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.044581 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:13:07.044763 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.044641 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:13:07.046438 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.046409 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm"] Nov 02 15:13:07.099814 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.099723 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5pm\" (UniqueName: \"kubernetes.io/projected/65ffd0af-55cc-4863-9739-15546b597534-kube-api-access-6x5pm\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.099814 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.099768 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ffd0af-55cc-4863-9739-15546b597534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.100036 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.099908 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ffd0af-55cc-4863-9739-15546b597534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.201420 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.201384 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ffd0af-55cc-4863-9739-15546b597534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.201593 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.201440 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5pm\" (UniqueName: \"kubernetes.io/projected/65ffd0af-55cc-4863-9739-15546b597534-kube-api-access-6x5pm\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.201593 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.201477 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ffd0af-55cc-4863-9739-15546b597534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.202208 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.202186 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ffd0af-55cc-4863-9739-15546b597534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.203642 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.203612 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ffd0af-55cc-4863-9739-15546b597534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.224482 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.224456 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5pm\" (UniqueName: \"kubernetes.io/projected/65ffd0af-55cc-4863-9739-15546b597534-kube-api-access-6x5pm\") pod \"maas-api-revoke-sub-3d4156f1-8jrkm\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.340459 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.340406 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:07.430783 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.430745 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df96964-40b2-4547-9328-850166719c72" path="/var/lib/kubelet/pods/5df96964-40b2-4547-9328-850166719c72/volumes" Nov 02 15:13:07.639777 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.639750 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm"] Nov 02 15:13:07.641867 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:13:07.641831 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ffd0af_55cc_4863_9739_15546b597534.slice/crio-477a3f8065381e54e033c46eac3b91b9c79b8501b4f73a3faa295476951341c1 WatchSource:0}: Error finding container 477a3f8065381e54e033c46eac3b91b9c79b8501b4f73a3faa295476951341c1: Status 404 returned error can't find the container with id 477a3f8065381e54e033c46eac3b91b9c79b8501b4f73a3faa295476951341c1 Nov 02 15:13:07.934836 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.934749 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" event={"ID":"65ffd0af-55cc-4863-9739-15546b597534","Type":"ContainerStarted","Data":"2e4b318fa4c4f6d85020523145aa5640bd0248add043243e7d592a049d8124c7"} Nov 02 15:13:07.934836 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.934819 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" event={"ID":"65ffd0af-55cc-4863-9739-15546b597534","Type":"ContainerStarted","Data":"477a3f8065381e54e033c46eac3b91b9c79b8501b4f73a3faa295476951341c1"} Nov 02 15:13:07.975699 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:07.975010 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" podStartSLOduration=0.974969766 podStartE2EDuration="974.969766ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:07.974425298 +0000 UTC m=+5336.700047068" watchObservedRunningTime="2026-11-02 15:13:07.974969766 +0000 UTC m=+5336.700591538" Nov 02 15:13:08.941439 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:08.941407 5591 generic.go:356] "Generic (PLEG): container finished" podID="65ffd0af-55cc-4863-9739-15546b597534" containerID="2e4b318fa4c4f6d85020523145aa5640bd0248add043243e7d592a049d8124c7" exitCode=0 Nov 02 15:13:08.941847 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:08.941493 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" event={"ID":"65ffd0af-55cc-4863-9739-15546b597534","Type":"ContainerDied","Data":"2e4b318fa4c4f6d85020523145aa5640bd0248add043243e7d592a049d8124c7"} Nov 02 15:13:10.073296 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.073272 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:10.228531 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.228427 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ffd0af-55cc-4863-9739-15546b597534-kube-api-access-6x5pm\" (UniqueName: \"kubernetes.io/projected/65ffd0af-55cc-4863-9739-15546b597534-kube-api-access-6x5pm\") pod \"65ffd0af-55cc-4863-9739-15546b597534\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " Nov 02 15:13:10.228531 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.228517 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ffd0af-55cc-4863-9739-15546b597534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ffd0af-55cc-4863-9739-15546b597534-maas-api-xks-ca\") pod \"65ffd0af-55cc-4863-9739-15546b597534\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " Nov 02 15:13:10.228799 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.228618 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ffd0af-55cc-4863-9739-15546b597534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ffd0af-55cc-4863-9739-15546b597534-maas-api-openshift-ca\") pod \"65ffd0af-55cc-4863-9739-15546b597534\" (UID: \"65ffd0af-55cc-4863-9739-15546b597534\") " Nov 02 15:13:10.229002 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.228952 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ffd0af-55cc-4863-9739-15546b597534-maas-api-openshift-ca" pod "65ffd0af-55cc-4863-9739-15546b597534" (UID: "65ffd0af-55cc-4863-9739-15546b597534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:13:10.230741 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.230710 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ffd0af-55cc-4863-9739-15546b597534-maas-api-xks-ca" pod "65ffd0af-55cc-4863-9739-15546b597534" (UID: "65ffd0af-55cc-4863-9739-15546b597534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:13:10.230741 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.230727 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ffd0af-55cc-4863-9739-15546b597534-kube-api-access-6x5pm" pod "65ffd0af-55cc-4863-9739-15546b597534" (UID: "65ffd0af-55cc-4863-9739-15546b597534"). InnerVolumeSpecName "kube-api-access-6x5pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:13:10.330231 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.330185 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ffd0af-55cc-4863-9739-15546b597534-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:10.330231 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.330224 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x5pm\" (UniqueName: \"kubernetes.io/projected/65ffd0af-55cc-4863-9739-15546b597534-kube-api-access-6x5pm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:10.330231 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.330234 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ffd0af-55cc-4863-9739-15546b597534-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:10.951854 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.951822 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" Nov 02 15:13:10.951854 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.951841 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm" event={"ID":"65ffd0af-55cc-4863-9739-15546b597534","Type":"ContainerDied","Data":"477a3f8065381e54e033c46eac3b91b9c79b8501b4f73a3faa295476951341c1"} Nov 02 15:13:10.952108 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:10.951886 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="477a3f8065381e54e033c46eac3b91b9c79b8501b4f73a3faa295476951341c1" Nov 02 15:13:15.179573 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:15.179538 5591 scope.go:122] "RemoveContainer" containerID="5a782203577c50f7dede64148173736bf3eea993dbb050e76df4b09f3fc76898" Nov 02 15:13:15.188195 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:15.188171 5591 scope.go:122] "RemoveContainer" containerID="d6d6d56eea0f3b8e54bb7b4c43c1fc116672e77ee8be2dd2a1c348d58c797d7a" Nov 02 15:13:15.196239 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:15.196220 5591 scope.go:122] "RemoveContainer" containerID="1c6939ab959303bb7e9f155ef7d299fe854513ec68bb3535cd95b76bf02b5197" Nov 02 15:13:19.229666 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:19.229631 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:13:24.640543 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:24.640498 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:13:30.350950 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:30.350909 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:13:30.948125 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:30.948078 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:31.075031 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:13:31.074968 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:13:31.240188 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:31.240095 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:36.493838 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.493799 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx"] Nov 02 15:13:36.494266 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.494239 5591 cpu_manager.go:414] "containerMap: removing container" podUID="65ffd0af-55cc-4863-9739-15546b597534" containerName="revoke-keys" Nov 02 15:13:36.494266 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.494254 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ffd0af-55cc-4863-9739-15546b597534" containerName="revoke-keys" Nov 02 15:13:36.494357 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.494346 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ffd0af-55cc-4863-9739-15546b597534" containerName="revoke-keys" Nov 02 15:13:36.496690 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.496664 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.511385 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.511351 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:13:36.511385 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.511374 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:13:36.523453 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.523422 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:13:36.523453 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.523448 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:13:36.525565 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.525534 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx"] Nov 02 15:13:36.638202 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.638161 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z259j\" (UniqueName: \"kubernetes.io/projected/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-kube-api-access-z259j\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.638399 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.638288 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.638399 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.638336 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.738995 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.738943 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.739210 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.739020 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.739210 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.739119 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z259j\" (UniqueName: \"kubernetes.io/projected/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-kube-api-access-z259j\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.739772 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.739742 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.741327 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.741307 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.762702 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.762643 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z259j\" (UniqueName: \"kubernetes.io/projected/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-kube-api-access-z259j\") pod \"maas-api-revoke-sub-e2fc9c5b-szlmx\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:36.810201 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:36.810172 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:37.106074 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:37.106050 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx"] Nov 02 15:13:37.108587 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:13:37.108553 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2221d9bd_e508_49a0_91f2_4520d2f1eaf5.slice/crio-b7fcfb2324101e574352cbf6e78f4f0103e944e2aac98a0770cb43a0ceca99e1 WatchSource:0}: Error finding container b7fcfb2324101e574352cbf6e78f4f0103e944e2aac98a0770cb43a0ceca99e1: Status 404 returned error can't find the container with id b7fcfb2324101e574352cbf6e78f4f0103e944e2aac98a0770cb43a0ceca99e1 Nov 02 15:13:38.115772 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:38.115728 5591 generic.go:356] "Generic (PLEG): container finished" podID="2221d9bd-e508-49a0-91f2-4520d2f1eaf5" containerID="f51d355ff8eb099b9f1c85a1e090ada8d4110a09c152e56140617737d7356e4f" exitCode=0 Nov 02 15:13:38.116208 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:38.115771 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" event={"ID":"2221d9bd-e508-49a0-91f2-4520d2f1eaf5","Type":"ContainerDied","Data":"f51d355ff8eb099b9f1c85a1e090ada8d4110a09c152e56140617737d7356e4f"} Nov 02 15:13:38.116208 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:38.115842 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" event={"ID":"2221d9bd-e508-49a0-91f2-4520d2f1eaf5","Type":"ContainerStarted","Data":"b7fcfb2324101e574352cbf6e78f4f0103e944e2aac98a0770cb43a0ceca99e1"} Nov 02 15:13:39.250600 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.250574 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:39.261661 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.261635 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-kube-api-access-z259j\" (UniqueName: \"kubernetes.io/projected/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-kube-api-access-z259j\") pod \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " Nov 02 15:13:39.261807 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.261671 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-xks-ca\") pod \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " Nov 02 15:13:39.261807 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.261714 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-openshift-ca\") pod \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\" (UID: \"2221d9bd-e508-49a0-91f2-4520d2f1eaf5\") " Nov 02 15:13:39.262160 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.262133 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-openshift-ca" pod "2221d9bd-e508-49a0-91f2-4520d2f1eaf5" (UID: "2221d9bd-e508-49a0-91f2-4520d2f1eaf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:13:39.263637 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.263613 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-kube-api-access-z259j" pod "2221d9bd-e508-49a0-91f2-4520d2f1eaf5" (UID: "2221d9bd-e508-49a0-91f2-4520d2f1eaf5"). InnerVolumeSpecName "kube-api-access-z259j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:13:39.263739 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.263687 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-xks-ca" pod "2221d9bd-e508-49a0-91f2-4520d2f1eaf5" (UID: "2221d9bd-e508-49a0-91f2-4520d2f1eaf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:13:39.362786 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.362747 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z259j\" (UniqueName: \"kubernetes.io/projected/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-kube-api-access-z259j\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:39.362786 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.362786 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:39.363032 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:39.362800 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2221d9bd-e508-49a0-91f2-4520d2f1eaf5-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:40.127203 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:40.127172 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" Nov 02 15:13:40.127397 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:40.127203 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx" event={"ID":"2221d9bd-e508-49a0-91f2-4520d2f1eaf5","Type":"ContainerDied","Data":"b7fcfb2324101e574352cbf6e78f4f0103e944e2aac98a0770cb43a0ceca99e1"} Nov 02 15:13:40.127397 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:40.127250 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7fcfb2324101e574352cbf6e78f4f0103e944e2aac98a0770cb43a0ceca99e1" Nov 02 15:13:46.816395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.816359 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t"] Nov 02 15:13:46.816878 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.816732 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2221d9bd-e508-49a0-91f2-4520d2f1eaf5" containerName="revoke-keys" Nov 02 15:13:46.816878 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.816743 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2221d9bd-e508-49a0-91f2-4520d2f1eaf5" containerName="revoke-keys" Nov 02 15:13:46.816878 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.816818 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2221d9bd-e508-49a0-91f2-4520d2f1eaf5" containerName="revoke-keys" Nov 02 15:13:46.820422 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.820398 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:46.840725 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.840694 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t"] Nov 02 15:13:46.850418 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.850383 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:13:46.850418 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.850397 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:13:46.850418 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.850391 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:13:46.850626 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.850438 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:13:46.933260 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.933224 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:46.933436 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.933270 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:46.933436 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:46.933346 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8sh\" (UniqueName: \"kubernetes.io/projected/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-kube-api-access-sg8sh\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:47.034234 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.034196 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:47.034410 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.034256 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:47.034410 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.034296 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8sh\" (UniqueName: \"kubernetes.io/projected/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-kube-api-access-sg8sh\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:47.034888 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.034862 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:47.036662 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.036642 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:47.057248 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.057224 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8sh\" (UniqueName: \"kubernetes.io/projected/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-kube-api-access-sg8sh\") pod \"maas-api-revoke-sub-5a29efb3-ct64t\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:47.134913 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.134885 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:47.438601 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.438539 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t"] Nov 02 15:13:47.441136 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:13:47.441106 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216b9755_6d9f_4f9e_b6e6_1f621a1fccd3.slice/crio-a43e4998f66d5c37c1cd5464679f51203c08e41fa823c2b053b67188c6f7408a WatchSource:0}: Error finding container a43e4998f66d5c37c1cd5464679f51203c08e41fa823c2b053b67188c6f7408a: Status 404 returned error can't find the container with id a43e4998f66d5c37c1cd5464679f51203c08e41fa823c2b053b67188c6f7408a Nov 02 15:13:47.851294 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:47.851259 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:13:48.166176 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:48.166134 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" event={"ID":"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3","Type":"ContainerStarted","Data":"4a13f1b6777ffec8735bcad680408d076eac0786e88bbb98cfa660dfe84655b9"} Nov 02 15:13:48.166360 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:48.166192 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" event={"ID":"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3","Type":"ContainerStarted","Data":"a43e4998f66d5c37c1cd5464679f51203c08e41fa823c2b053b67188c6f7408a"} Nov 02 15:13:48.207694 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:48.207646 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" podStartSLOduration=2.207630391 podStartE2EDuration="2.207630391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:48.206309333 +0000 UTC m=+5376.931931103" watchObservedRunningTime="2026-11-02 15:13:48.207630391 +0000 UTC m=+5376.933252164" Nov 02 15:13:49.198251 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:49.198210 5591 generic.go:356] "Generic (PLEG): container finished" podID="216b9755-6d9f-4f9e-b6e6-1f621a1fccd3" containerID="4a13f1b6777ffec8735bcad680408d076eac0786e88bbb98cfa660dfe84655b9" exitCode=0 Nov 02 15:13:49.202391 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:49.198809 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" event={"ID":"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3","Type":"ContainerDied","Data":"4a13f1b6777ffec8735bcad680408d076eac0786e88bbb98cfa660dfe84655b9"} Nov 02 15:13:50.552208 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.552171 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:50.624953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.624914 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-xks-ca\") pod \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " Nov 02 15:13:50.625170 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.625014 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-openshift-ca\") pod \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " Nov 02 15:13:50.625170 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.625048 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-kube-api-access-sg8sh\" (UniqueName: \"kubernetes.io/projected/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-kube-api-access-sg8sh\") pod \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\" (UID: \"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3\") " Nov 02 15:13:50.625390 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.625362 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-openshift-ca" pod "216b9755-6d9f-4f9e-b6e6-1f621a1fccd3" (UID: "216b9755-6d9f-4f9e-b6e6-1f621a1fccd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:13:50.627192 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.627161 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-xks-ca" pod "216b9755-6d9f-4f9e-b6e6-1f621a1fccd3" (UID: "216b9755-6d9f-4f9e-b6e6-1f621a1fccd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:13:50.627319 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.627294 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-kube-api-access-sg8sh" pod "216b9755-6d9f-4f9e-b6e6-1f621a1fccd3" (UID: "216b9755-6d9f-4f9e-b6e6-1f621a1fccd3"). InnerVolumeSpecName "kube-api-access-sg8sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:13:50.726053 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.726012 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:50.726053 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.726056 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:50.726293 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.726074 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg8sh\" (UniqueName: \"kubernetes.io/projected/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3-kube-api-access-sg8sh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:13:50.798460 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.798329 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:13:50.798460 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:50.798407 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:13:51.213504 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:51.213467 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" event={"ID":"216b9755-6d9f-4f9e-b6e6-1f621a1fccd3","Type":"ContainerDied","Data":"a43e4998f66d5c37c1cd5464679f51203c08e41fa823c2b053b67188c6f7408a"} Nov 02 15:13:51.213504 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:51.213503 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t" Nov 02 15:13:51.213730 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:51.213522 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a43e4998f66d5c37c1cd5464679f51203c08e41fa823c2b053b67188c6f7408a" Nov 02 15:13:58.145325 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:13:58.145291 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:14:01.072275 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:14:01.072239 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:14:04.054790 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:04.054749 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:14:12.009390 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:12.009350 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:14:12.448745 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:12.448711 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:14:12.457929 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:12.457899 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:14:30.223740 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:30.223649 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:31.072827 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:14:31.072791 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:14:31.846604 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:31.846574 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:33.712302 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:33.712263 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:34.075094 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.075004 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h"] Nov 02 15:14:34.075674 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.075650 5591 cpu_manager.go:414] "containerMap: removing container" podUID="216b9755-6d9f-4f9e-b6e6-1f621a1fccd3" containerName="revoke-keys" Nov 02 15:14:34.075726 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.075678 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216b9755-6d9f-4f9e-b6e6-1f621a1fccd3" containerName="revoke-keys" Nov 02 15:14:34.075802 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.075789 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="216b9755-6d9f-4f9e-b6e6-1f621a1fccd3" containerName="revoke-keys" Nov 02 15:14:34.078994 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.078961 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.093528 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.093500 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-437128-openshift-default-dockercfg-pjts4\"" Nov 02 15:14:34.108756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.108717 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h"] Nov 02 15:14:34.185402 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185115 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:34.185612 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185519 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-socket\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.185612 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185574 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-podinfo\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.185612 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185608 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xm24\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-kube-api-access-5xm24\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.185818 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185660 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-token\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.185818 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185700 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-credential-socket\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.185818 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185726 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-certs\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.185818 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185788 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-envoy\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.185818 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185814 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0266d5e-3e55-403c-ae7e-001d058ce705-istiod-ca-cert\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.186166 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.185844 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-data\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287227 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287186 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-socket\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287466 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287236 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-podinfo\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287466 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287257 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xm24\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-kube-api-access-5xm24\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287466 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287298 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-token\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287466 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287322 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-credential-socket\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287466 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287346 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-certs\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287771 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287516 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-envoy\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287771 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287563 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0266d5e-3e55-403c-ae7e-001d058ce705-istiod-ca-cert\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287771 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287595 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-data\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287771 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287675 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-socket\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.287771 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287733 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-credential-socket\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.288017 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.287843 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-certs\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.288063 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.288021 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-data\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.288329 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.288311 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0266d5e-3e55-403c-ae7e-001d058ce705-istiod-ca-cert\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.289670 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.289645 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-envoy\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.289780 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.289762 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-podinfo\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.310363 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.310336 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-token\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.310488 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.310413 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xm24\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-kube-api-access-5xm24\") pod \"e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.394012 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.393961 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:34.417645 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.417613 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:34.720470 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.720439 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h"] Nov 02 15:14:34.722599 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:14:34.722570 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0266d5e_3e55_403c_ae7e_001d058ce705.slice/crio-b76c106418e065ed9765431bb3c7fb3254feb925e58dc7aca061cab736eb5a61 WatchSource:0}: Error finding container b76c106418e065ed9765431bb3c7fb3254feb925e58dc7aca061cab736eb5a61: Status 404 returned error can't find the container with id b76c106418e065ed9765431bb3c7fb3254feb925e58dc7aca061cab736eb5a61 Nov 02 15:14:34.724867 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.724834 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:14:34.724933 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.724902 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:14:34.724977 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:34.724930 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:14:35.447079 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:35.447039 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" event={"ID":"f0266d5e-3e55-403c-ae7e-001d058ce705","Type":"ContainerStarted","Data":"450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d"} Nov 02 15:14:35.447281 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:35.447104 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" event={"ID":"f0266d5e-3e55-403c-ae7e-001d058ce705","Type":"ContainerStarted","Data":"b76c106418e065ed9765431bb3c7fb3254feb925e58dc7aca061cab736eb5a61"} Nov 02 15:14:35.510800 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:35.510740 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podStartSLOduration=1.510727395 podStartE2EDuration="1.510727395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:35.50864539 +0000 UTC m=+5424.234267161" watchObservedRunningTime="2026-11-02 15:14:35.510727395 +0000 UTC m=+5424.236349164" Nov 02 15:14:35.941288 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:35.941249 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:36.394547 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:36.394511 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:36.399682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:36.399651 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:36.452062 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:36.452029 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:36.453037 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:36.453016 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:37.949338 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:37.949303 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:38.126398 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:38.126359 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:39.254241 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.254204 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r"] Nov 02 15:14:39.257919 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.257895 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.273151 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.273123 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-437128-serving-cert\"" Nov 02 15:14:39.273300 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.273280 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:14:39.279071 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.279042 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 15:14:39.285428 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.285392 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:14:39.287471 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.287442 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r"] Nov 02 15:14:39.289075 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.289045 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:14:39.338963 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.338928 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae00a5f-7116-455b-9bb2-75d01effe58c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.339166 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.339053 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.339166 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.339086 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.339296 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.339233 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdj5\" (UniqueName: \"kubernetes.io/projected/bae00a5f-7116-455b-9bb2-75d01effe58c-kube-api-access-zfdj5\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.390858 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.390821 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:39.440536 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.440496 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae00a5f-7116-455b-9bb2-75d01effe58c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.440718 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.440567 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.440718 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.440606 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.440718 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.440657 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdj5\" (UniqueName: \"kubernetes.io/projected/bae00a5f-7116-455b-9bb2-75d01effe58c-kube-api-access-zfdj5\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.441333 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.441304 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae00a5f-7116-455b-9bb2-75d01effe58c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.443174 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.443150 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.443376 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.443353 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.468962 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.468845 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdj5\" (UniqueName: \"kubernetes.io/projected/bae00a5f-7116-455b-9bb2-75d01effe58c-kube-api-access-zfdj5\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.557749 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.557646 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:39.572498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.572460 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:39.727667 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.727632 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:39.874607 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.874582 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r"] Nov 02 15:14:39.877231 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:14:39.877194 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae00a5f_7116_455b_9bb2_75d01effe58c.slice/crio-156bb3531274876c65ef7423655231fc6fab4696e23846d461c5fb00211b7d92 WatchSource:0}: Error finding container 156bb3531274876c65ef7423655231fc6fab4696e23846d461c5fb00211b7d92: Status 404 returned error can't find the container with id 156bb3531274876c65ef7423655231fc6fab4696e23846d461c5fb00211b7d92 Nov 02 15:14:39.878576 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:39.878554 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:14:40.473880 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:40.473845 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" event={"ID":"bae00a5f-7116-455b-9bb2-75d01effe58c","Type":"ContainerStarted","Data":"faab284ebd84856af358b335905def8c87accf768848793d8494454adc8b32eb"} Nov 02 15:14:40.474407 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:40.473901 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" event={"ID":"bae00a5f-7116-455b-9bb2-75d01effe58c","Type":"ContainerStarted","Data":"156bb3531274876c65ef7423655231fc6fab4696e23846d461c5fb00211b7d92"} Nov 02 15:14:40.474407 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:40.473978 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:40.516194 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:40.516138 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" podStartSLOduration=1.011925876 podStartE2EDuration="1.516124946s" totalImagesPullingTime="504.19907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:40.514863773 +0000 UTC m=+5429.240485521" watchObservedRunningTime="2026-11-02 15:14:40.516124946 +0000 UTC m=+5429.241746716" Nov 02 15:14:40.928289 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:40.928150 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:42.498833 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:42.498790 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:43.015594 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:43.015560 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:14:43.962573 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:43.962537 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:45.642702 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:45.642644 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:46.484185 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:46.484146 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:14:47.290630 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:47.290589 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:47.723119 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:47.723085 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:48.326677 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:48.326637 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:48.502460 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:48.502426 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:48.687776 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:48.687739 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:48.960391 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:48.960309 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:49.781061 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.781017 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk"] Nov 02 15:14:49.785751 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.785724 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.816776 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.816740 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-437128\"/\"openshift-service-ca.crt\"" Nov 02 15:14:49.816776 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.816740 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-437128\"/\"kube-root-ca.crt\"" Nov 02 15:14:49.817017 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.816803 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-437128\"/\"simulator-single-main-437128-kserve-self-signed-certs\"" Nov 02 15:14:49.817017 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.816809 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-437128\"/\"default-dockercfg-4bl42\"" Nov 02 15:14:49.819873 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.819841 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk"] Nov 02 15:14:49.838649 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.838603 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-model-cache\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.838837 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.838721 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-home\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.838837 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.838774 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fcw\" (UniqueName: \"kubernetes.io/projected/65dfdc8c-0bf1-4914-8065-83dd3fd42044-kube-api-access-k6fcw\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.838837 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.838811 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tls-certs\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.838837 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.838826 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-dshm\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.839107 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.838901 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tmp-dir\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.903240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.903207 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:49.940225 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940186 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-model-cache\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.940433 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940246 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-home\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.940433 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940281 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fcw\" (UniqueName: \"kubernetes.io/projected/65dfdc8c-0bf1-4914-8065-83dd3fd42044-kube-api-access-k6fcw\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.940433 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940333 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tls-certs\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.940433 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940357 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-dshm\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.940433 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940382 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tmp-dir\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.940757 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940708 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-home\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.940757 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940726 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-model-cache\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.940963 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.940943 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tmp-dir\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.942631 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.942603 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-dshm\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.943350 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.943329 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tls-certs\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:49.965742 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:49.965706 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fcw\" (UniqueName: \"kubernetes.io/projected/65dfdc8c-0bf1-4914-8065-83dd3fd42044-kube-api-access-k6fcw\") pod \"simulator-single-main-437128-kserve-657f877cb9-4qgqk\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:50.116798 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:50.116674 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:50.428592 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:50.426188 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk"] Nov 02 15:14:50.428953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:50.428748 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:50.432871 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:14:50.432841 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65dfdc8c_0bf1_4914_8065_83dd3fd42044.slice/crio-a9a9302b190bdd87f6f18df3fe94612a1a0a4e77dca0f7659e94220bf73f1796 WatchSource:0}: Error finding container a9a9302b190bdd87f6f18df3fe94612a1a0a4e77dca0f7659e94220bf73f1796: Status 404 returned error can't find the container with id a9a9302b190bdd87f6f18df3fe94612a1a0a4e77dca0f7659e94220bf73f1796 Nov 02 15:14:50.571286 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:50.571247 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" event={"ID":"65dfdc8c-0bf1-4914-8065-83dd3fd42044","Type":"ContainerStarted","Data":"7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681"} Nov 02 15:14:50.571489 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:50.571296 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" event={"ID":"65dfdc8c-0bf1-4914-8065-83dd3fd42044","Type":"ContainerStarted","Data":"a9a9302b190bdd87f6f18df3fe94612a1a0a4e77dca0f7659e94220bf73f1796"} Nov 02 15:14:50.613198 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:50.613119 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" podStartSLOduration=1.613099403 podStartE2EDuration="1.613099403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:50.611575737 +0000 UTC m=+5439.337197497" watchObservedRunningTime="2026-11-02 15:14:50.613099403 +0000 UTC m=+5439.338721176" Nov 02 15:14:51.117059 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:51.117017 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:14:51.936849 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:51.936806 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:52.135874 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.135832 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:14:52.173568 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.173449 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": read tcp 10.133.0.2:36740->10.133.1.89:15021: read: connection reset by peer" start-of-body= Nov 02 15:14:52.173568 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.173533 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": read tcp 10.133.0.2:36740->10.133.1.89:15021: read: connection reset by peer" Nov 02 15:14:52.583463 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.583369 5591 generic.go:356] "Generic (PLEG): container finished" podID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerID="450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d" exitCode=0 Nov 02 15:14:52.583463 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.583446 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" event={"ID":"f0266d5e-3e55-403c-ae7e-001d058ce705","Type":"ContainerDied","Data":"450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d"} Nov 02 15:14:52.583929 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.583915 5591 scope.go:122] "RemoveContainer" containerID="450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d" Nov 02 15:14:52.585124 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.585097 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:14:52.585206 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.585166 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:14:52.585206 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:52.585197 5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:14:53.557571 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:53.557532 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:53.590739 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:53.590703 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" event={"ID":"f0266d5e-3e55-403c-ae7e-001d058ce705","Type":"ContainerStarted","Data":"0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb"} Nov 02 15:14:54.395125 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:54.395093 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:54.396023 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:54.395473 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 15:14:54.396023 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:54.395542 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 15:14:55.106140 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:55.106105 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:55.394900 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:55.394862 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 15:14:55.395112 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:55.394923 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 15:14:56.395004 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:56.394950 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 15:14:56.395440 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:56.395019 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 15:14:56.713545 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:56.713460 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:57.395036 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:57.394978 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 15:14:57.395682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:57.395633 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 15:14:58.246155 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:58.246117 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:58.394911 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:58.394881 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Nov 02 15:14:58.395136 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:58.394941 5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Nov 02 15:14:59.398665 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:59.398625 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:59.399186 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:59.399018 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:59.399657 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:59.399636 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:14:59.798319 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:14:59.798217 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:00.028480 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:00.028442 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg"] Nov 02 15:15:00.031149 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:00.031129 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pnghg"] Nov 02 15:15:00.117454 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:00.117350 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:15:00.130355 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:00.130324 5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:15:00.636511 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:00.636484 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:15:01.073942 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:15:01.073849 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:15:01.208519 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:01.208440 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:01.437508 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:01.437467 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d119eef7-761c-49a1-8295-1b4b3406ceee" path="/var/lib/kubelet/pods/d119eef7-761c-49a1-8295-1b4b3406ceee/volumes" Nov 02 15:15:02.608574 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:02.608523 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:04.007070 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:04.007034 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:05.415806 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:05.415768 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:06.914127 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:06.913951 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:08.343375 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:08.343337 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:09.540822 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:09.540756 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:15:09.853821 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:09.853710 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:11.321051 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:11.321014 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:12.720263 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:12.720225 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:14.120893 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:14.120856 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:15.522266 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:15.522231 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:17.004730 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:17.004688 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:18.418240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:18.418201 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:19.810638 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:19.810579 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:21.207430 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:21.207395 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:22.039042 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:22.039004 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp"] Nov 02 15:15:22.051114 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:22.051075 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpmdp"] Nov 02 15:15:22.612078 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:22.612020 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:23.428239 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:23.428201 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2" path="/var/lib/kubelet/pods/7ced1d0c-07b3-420c-bf3d-0b8e5ed7bef2/volumes" Nov 02 15:15:24.110689 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:24.110655 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:25.621736 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:25.621692 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:27.007756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:27.007710 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:28.533953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:28.532438 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:30.023099 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:30.023027 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:31.073399 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:15:31.073360 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:15:31.177889 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:31.177855 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:31.341223 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:31.341144 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:31.535493 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:31.532887 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:33.013191 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:33.013155 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:34.413060 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:34.413013 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:35.817603 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:35.817560 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:37.218004 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:37.217951 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:38.596335 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:38.596282 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:38.759618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:38.759576 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:39.058236 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:39.058203 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:40.039433 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:40.039384 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:15:40.118227 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:40.118193 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:41.616893 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:41.616803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:43.029100 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:43.029059 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4"] Nov 02 15:15:43.040570 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:43.040526 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9rzj4"] Nov 02 15:15:43.127331 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:43.127293 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:43.428036 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:43.428005 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f945e8bb-6227-4304-a101-961d33363966" path="/var/lib/kubelet/pods/f945e8bb-6227-4304-a101-961d33363966/volumes" Nov 02 15:15:43.509752 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:43.509716 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk"] Nov 02 15:15:43.689271 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:43.689183 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:44.713051 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:44.713017 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:44.876351 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:44.876305 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:45.212532 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:45.210284 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:46.308618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:46.308581 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:47.419527 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:47.419481 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r"] Nov 02 15:15:47.419952 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:47.419780 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" podUID="bae00a5f-7116-455b-9bb2-75d01effe58c" containerName="maas-api" containerID="cri-o://faab284ebd84856af358b335905def8c87accf768848793d8494454adc8b32eb" gracePeriod=30 Nov 02 15:15:47.547505 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:47.547468 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:47.885314 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:47.885283 5591 generic.go:356] "Generic (PLEG): container finished" podID="bae00a5f-7116-455b-9bb2-75d01effe58c" containerID="faab284ebd84856af358b335905def8c87accf768848793d8494454adc8b32eb" exitCode=0 Nov 02 15:15:47.885477 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:47.885354 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" event={"ID":"bae00a5f-7116-455b-9bb2-75d01effe58c","Type":"ContainerDied","Data":"faab284ebd84856af358b335905def8c87accf768848793d8494454adc8b32eb"} Nov 02 15:15:47.885477 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:47.885407 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" event={"ID":"bae00a5f-7116-455b-9bb2-75d01effe58c","Type":"ContainerDied","Data":"156bb3531274876c65ef7423655231fc6fab4696e23846d461c5fb00211b7d92"} Nov 02 15:15:47.885477 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:47.885427 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="156bb3531274876c65ef7423655231fc6fab4696e23846d461c5fb00211b7d92" Nov 02 15:15:47.895160 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:47.895141 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:15:48.083862 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.083780 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae00a5f-7116-455b-9bb2-75d01effe58c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae00a5f-7116-455b-9bb2-75d01effe58c-openshift-service-ca\") pod \"bae00a5f-7116-455b-9bb2-75d01effe58c\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " Nov 02 15:15:48.083862 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.083829 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-metrics-certs\") pod \"bae00a5f-7116-455b-9bb2-75d01effe58c\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " Nov 02 15:15:48.084116 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.083890 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-maas-api-tls\") pod \"bae00a5f-7116-455b-9bb2-75d01effe58c\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " Nov 02 15:15:48.084116 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.083948 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae00a5f-7116-455b-9bb2-75d01effe58c-kube-api-access-zfdj5\" (UniqueName: \"kubernetes.io/projected/bae00a5f-7116-455b-9bb2-75d01effe58c-kube-api-access-zfdj5\") pod \"bae00a5f-7116-455b-9bb2-75d01effe58c\" (UID: \"bae00a5f-7116-455b-9bb2-75d01effe58c\") " Nov 02 15:15:48.084274 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.084246 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae00a5f-7116-455b-9bb2-75d01effe58c-openshift-service-ca" pod "bae00a5f-7116-455b-9bb2-75d01effe58c" (UID: "bae00a5f-7116-455b-9bb2-75d01effe58c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:48.086181 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.086146 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-maas-api-tls" pod "bae00a5f-7116-455b-9bb2-75d01effe58c" (UID: "bae00a5f-7116-455b-9bb2-75d01effe58c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:15:48.086331 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.086179 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-metrics-certs" pod "bae00a5f-7116-455b-9bb2-75d01effe58c" (UID: "bae00a5f-7116-455b-9bb2-75d01effe58c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:15:48.086331 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.086244 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae00a5f-7116-455b-9bb2-75d01effe58c-kube-api-access-zfdj5" pod "bae00a5f-7116-455b-9bb2-75d01effe58c" (UID: "bae00a5f-7116-455b-9bb2-75d01effe58c"). InnerVolumeSpecName "kube-api-access-zfdj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:48.184946 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.184904 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:48.184946 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.184940 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfdj5\" (UniqueName: \"kubernetes.io/projected/bae00a5f-7116-455b-9bb2-75d01effe58c-kube-api-access-zfdj5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:48.184946 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.184952 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae00a5f-7116-455b-9bb2-75d01effe58c-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:48.185232 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.184961 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bae00a5f-7116-455b-9bb2-75d01effe58c-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:48.890645 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.890608 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r" Nov 02 15:15:48.941441 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.941407 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r"] Nov 02 15:15:48.953307 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:48.953277 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-kcr8r"] Nov 02 15:15:49.429777 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:49.429743 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae00a5f-7116-455b-9bb2-75d01effe58c" path="/var/lib/kubelet/pods/bae00a5f-7116-455b-9bb2-75d01effe58c/volumes" Nov 02 15:15:49.912767 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:49.912733 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:15:50.964265 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:50.964224 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:51.139882 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:51.139841 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:54.039208 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:54.039127 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj"] Nov 02 15:15:54.051086 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:54.051049 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cjmdj"] Nov 02 15:15:55.428398 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:55.428359 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d1a927-1206-4b6b-852a-14bdd242e7c4" path="/var/lib/kubelet/pods/39d1a927-1206-4b6b-852a-14bdd242e7c4/volumes" Nov 02 15:15:58.530124 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.530077 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" podUID="65dfdc8c-0bf1-4914-8065-83dd3fd42044" containerName="main" containerID="cri-o://7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681" gracePeriod=45 Nov 02 15:15:58.625921 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.625873 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" podUID="65dfdc8c-0bf1-4914-8065-83dd3fd42044" containerName="main" probeResult="failure" output="Get \"https://10.133.1.91:8000/ready\": dial tcp 10.133.1.91:8000: connect: connection refused" Nov 02 15:15:58.775642 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.775614 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:15:58.876902 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.876817 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tls-certs\" (UniqueName: \"kubernetes.io/secret/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tls-certs\") pod \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " Nov 02 15:15:58.876902 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.876864 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-home\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-home\") pod \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " Nov 02 15:15:58.877144 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.876917 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dfdc8c-0bf1-4914-8065-83dd3fd42044-kube-api-access-k6fcw\" (UniqueName: \"kubernetes.io/projected/65dfdc8c-0bf1-4914-8065-83dd3fd42044-kube-api-access-k6fcw\") pod \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " Nov 02 15:15:58.877144 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.877008 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-model-cache\") pod \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " Nov 02 15:15:58.877144 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.877036 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tmp-dir\") pod \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " Nov 02 15:15:58.877144 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.877114 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-dshm\") pod \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\" (UID: \"65dfdc8c-0bf1-4914-8065-83dd3fd42044\") " Nov 02 15:15:58.877319 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.877226 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-home" pod "65dfdc8c-0bf1-4914-8065-83dd3fd42044" (UID: "65dfdc8c-0bf1-4914-8065-83dd3fd42044"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:58.877466 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.877393 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-model-cache" pod "65dfdc8c-0bf1-4914-8065-83dd3fd42044" (UID: "65dfdc8c-0bf1-4914-8065-83dd3fd42044"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:58.877543 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.877462 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tmp-dir" pod "65dfdc8c-0bf1-4914-8065-83dd3fd42044" (UID: "65dfdc8c-0bf1-4914-8065-83dd3fd42044"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:58.877543 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.877471 5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:58.879369 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.879338 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-dshm" pod "65dfdc8c-0bf1-4914-8065-83dd3fd42044" (UID: "65dfdc8c-0bf1-4914-8065-83dd3fd42044"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:58.879550 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.879530 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tls-certs" pod "65dfdc8c-0bf1-4914-8065-83dd3fd42044" (UID: "65dfdc8c-0bf1-4914-8065-83dd3fd42044"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:15:58.879622 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.879551 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dfdc8c-0bf1-4914-8065-83dd3fd42044-kube-api-access-k6fcw" pod "65dfdc8c-0bf1-4914-8065-83dd3fd42044" (UID: "65dfdc8c-0bf1-4914-8065-83dd3fd42044"). InnerVolumeSpecName "kube-api-access-k6fcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:58.942307 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.942272 5591 generic.go:356] "Generic (PLEG): container finished" podID="65dfdc8c-0bf1-4914-8065-83dd3fd42044" containerID="7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681" exitCode=0 Nov 02 15:15:58.942498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.942314 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" Nov 02 15:15:58.942498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.942318 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" event={"ID":"65dfdc8c-0bf1-4914-8065-83dd3fd42044","Type":"ContainerDied","Data":"7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681"} Nov 02 15:15:58.942498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.942442 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk" event={"ID":"65dfdc8c-0bf1-4914-8065-83dd3fd42044","Type":"ContainerDied","Data":"a9a9302b190bdd87f6f18df3fe94612a1a0a4e77dca0f7659e94220bf73f1796"} Nov 02 15:15:58.942498 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.942470 5591 scope.go:122] "RemoveContainer" containerID="7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681" Nov 02 15:15:58.954180 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.954159 5591 scope.go:122] "RemoveContainer" containerID="7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681" Nov 02 15:15:58.954494 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:15:58.954464 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681\": container with ID starting with 7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681 not found: ID does not exist" containerID="7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681" Nov 02 15:15:58.954569 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.954501 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681"} err="failed to get container status \"7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681\": rpc error: code = NotFound desc = could not find container \"7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681\": container with ID starting with 7c66f6967b58745225330a3c52ee2d5aeb91e2ad8637df2d539b49e8b4d03681 not found: ID does not exist" Nov 02 15:15:58.978728 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.978701 5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:58.978728 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.978723 5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:58.978883 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.978734 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6fcw\" (UniqueName: \"kubernetes.io/projected/65dfdc8c-0bf1-4914-8065-83dd3fd42044-kube-api-access-k6fcw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:58.978883 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.978743 5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:58.978883 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.978753 5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65dfdc8c-0bf1-4914-8065-83dd3fd42044-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:15:58.992979 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.992953 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk"] Nov 02 15:15:58.995564 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:58.995544 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-437128/simulator-single-main-437128-kserve-657f877cb9-4qgqk"] Nov 02 15:15:59.428172 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:15:59.428134 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dfdc8c-0bf1-4914-8065-83dd3fd42044" path="/var/lib/kubelet/pods/65dfdc8c-0bf1-4914-8065-83dd3fd42044/volumes" Nov 02 15:16:01.072084 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:16:01.072049 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:16:15.239889 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:15.239857 5591 scope.go:122] "RemoveContainer" containerID="0b701894b032af665c4d54ff8ad1b2c10aed8aa1494bef3d2746ac795d53dfa4" Nov 02 15:16:18.080906 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:18.080853 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:18.251794 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:18.251751 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:19.346679 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.346640 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6"] Nov 02 15:16:19.347133 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.347061 5591 cpu_manager.go:414] "containerMap: removing container" podUID="65dfdc8c-0bf1-4914-8065-83dd3fd42044" containerName="main" Nov 02 15:16:19.347133 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.347074 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dfdc8c-0bf1-4914-8065-83dd3fd42044" containerName="main" Nov 02 15:16:19.347133 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.347088 5591 cpu_manager.go:414] "containerMap: removing container" podUID="bae00a5f-7116-455b-9bb2-75d01effe58c" containerName="maas-api" Nov 02 15:16:19.347133 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.347094 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae00a5f-7116-455b-9bb2-75d01effe58c" containerName="maas-api" Nov 02 15:16:19.347292 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.347155 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dfdc8c-0bf1-4914-8065-83dd3fd42044" containerName="main" Nov 02 15:16:19.347292 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.347164 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae00a5f-7116-455b-9bb2-75d01effe58c" containerName="maas-api" Nov 02 15:16:19.352194 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.352170 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.374951 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.374910 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6"] Nov 02 15:16:19.381055 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.381017 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-437128-serving-cert\"" Nov 02 15:16:19.381055 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.381045 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:16:19.381316 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.381084 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:16:19.381316 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.381104 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:16:19.381476 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.381453 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 15:16:19.463145 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.463108 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bd792b-3787-4080-901b-b73ce0e7b8ea-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.463374 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.463157 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-metrics-certs\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.463374 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.463242 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.463374 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.463298 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4gwn\" (UniqueName: \"kubernetes.io/projected/72bd792b-3787-4080-901b-b73ce0e7b8ea-kube-api-access-l4gwn\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.481293 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.481253 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:19.564708 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.564657 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-metrics-certs\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.564914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.564711 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.564914 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.564874 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4gwn\" (UniqueName: \"kubernetes.io/projected/72bd792b-3787-4080-901b-b73ce0e7b8ea-kube-api-access-l4gwn\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.565094 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.565077 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bd792b-3787-4080-901b-b73ce0e7b8ea-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.565947 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.565895 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bd792b-3787-4080-901b-b73ce0e7b8ea-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.567649 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.567622 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-metrics-certs\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.567841 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.567819 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.590707 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.590667 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4gwn\" (UniqueName: \"kubernetes.io/projected/72bd792b-3787-4080-901b-b73ce0e7b8ea-kube-api-access-l4gwn\") pod \"maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.665936 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.665902 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:19.681533 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.681468 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:19.863271 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.863229 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:19.974936 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:19.974841 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6"] Nov 02 15:16:19.994357 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:16:19.993770 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bd792b_3787_4080_901b_b73ce0e7b8ea.slice/crio-5411dee8f550bdafc110f6ce05faf855527fd349d41a071dbf3494c197c33676 WatchSource:0}: Error finding container 5411dee8f550bdafc110f6ce05faf855527fd349d41a071dbf3494c197c33676: Status 404 returned error can't find the container with id 5411dee8f550bdafc110f6ce05faf855527fd349d41a071dbf3494c197c33676 Nov 02 15:16:20.051197 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:20.051159 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" event={"ID":"72bd792b-3787-4080-901b-b73ce0e7b8ea","Type":"ContainerStarted","Data":"5411dee8f550bdafc110f6ce05faf855527fd349d41a071dbf3494c197c33676"} Nov 02 15:16:20.844598 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:20.844556 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:21.038234 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:21.038202 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx"] Nov 02 15:16:21.049966 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:21.049937 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-sdcrx"] Nov 02 15:16:21.057293 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:21.057259 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" event={"ID":"72bd792b-3787-4080-901b-b73ce0e7b8ea","Type":"ContainerStarted","Data":"baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96"} Nov 02 15:16:21.057476 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:21.057458 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:21.100765 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:21.100644 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" podStartSLOduration=1.737528864 podStartE2EDuration="2.100630403s" totalImagesPullingTime="363.101539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:21.099411116 +0000 UTC m=+5529.825032890" watchObservedRunningTime="2026-11-02 15:16:21.100630403 +0000 UTC m=+5529.826252199" Nov 02 15:16:21.114814 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:21.114778 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:21.442409 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:21.442357 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7468325f-d3e1-48ef-9714-1060012b296b" path="/var/lib/kubelet/pods/7468325f-d3e1-48ef-9714-1060012b296b/volumes" Nov 02 15:16:22.066134 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:22.066102 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:22.783667 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:22.783617 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:23.244764 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:23.244726 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:16:24.323778 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:24.323739 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:25.809257 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:25.809217 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:27.068138 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:27.068107 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:27.215352 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:27.215316 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:28.283662 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:28.283619 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:28.453349 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:28.453292 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:28.731099 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:28.731045 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:30.177946 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:30.177908 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:31.028478 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:31.028441 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz"] Nov 02 15:16:31.040085 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:31.040059 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6snqz"] Nov 02 15:16:31.072973 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:16:31.072937 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:16:31.286777 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:31.286668 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:31.429402 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:31.429363 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e360a8-a775-468c-8603-2a8435a765d1" path="/var/lib/kubelet/pods/27e360a8-a775-468c-8603-2a8435a765d1/volumes" Nov 02 15:16:31.607029 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:31.606928 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:33.004470 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:33.004382 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:34.406425 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:34.406388 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:35.916399 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:35.916362 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:37.329144 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:37.329106 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:38.801180 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:38.801128 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:40.222267 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:40.222217 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:40.502718 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:40.502633 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:40.670662 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:40.670624 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:41.717692 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:41.717655 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:42.805387 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:42.805351 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6"] Nov 02 15:16:42.805815 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:42.805591 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" podUID="72bd792b-3787-4080-901b-b73ce0e7b8ea" containerName="maas-api" containerID="cri-o://baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96" gracePeriod=30 Nov 02 15:16:42.967406 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:42.967364 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:43.092026 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.091317 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:43.174342 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.174304 5591 generic.go:356] "Generic (PLEG): container finished" podID="72bd792b-3787-4080-901b-b73ce0e7b8ea" containerID="baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96" exitCode=0 Nov 02 15:16:43.174550 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.174352 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" Nov 02 15:16:43.174550 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.174383 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" event={"ID":"72bd792b-3787-4080-901b-b73ce0e7b8ea","Type":"ContainerDied","Data":"baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96"} Nov 02 15:16:43.174550 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.174455 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6" event={"ID":"72bd792b-3787-4080-901b-b73ce0e7b8ea","Type":"ContainerDied","Data":"5411dee8f550bdafc110f6ce05faf855527fd349d41a071dbf3494c197c33676"} Nov 02 15:16:43.174550 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.174489 5591 scope.go:122] "RemoveContainer" containerID="baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96" Nov 02 15:16:43.183658 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.183637 5591 scope.go:122] "RemoveContainer" containerID="baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96" Nov 02 15:16:43.183969 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:16:43.183940 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96\": container with ID starting with baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96 not found: ID does not exist" containerID="baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96" Nov 02 15:16:43.184057 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.183979 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96"} err="failed to get container status \"baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96\": rpc error: code = NotFound desc = could not find container \"baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96\": container with ID starting with baaaedbc6ad5c71d5fb4cc1e23e617e51cd75316013a8479ee82ce3817652e96 not found: ID does not exist" Nov 02 15:16:43.196514 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.196486 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72bd792b-3787-4080-901b-b73ce0e7b8ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bd792b-3787-4080-901b-b73ce0e7b8ea-openshift-service-ca\") pod \"72bd792b-3787-4080-901b-b73ce0e7b8ea\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " Nov 02 15:16:43.196618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.196562 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bd792b-3787-4080-901b-b73ce0e7b8ea-kube-api-access-l4gwn\" (UniqueName: \"kubernetes.io/projected/72bd792b-3787-4080-901b-b73ce0e7b8ea-kube-api-access-l4gwn\") pod \"72bd792b-3787-4080-901b-b73ce0e7b8ea\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " Nov 02 15:16:43.196699 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.196673 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-metrics-certs\") pod \"72bd792b-3787-4080-901b-b73ce0e7b8ea\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " Nov 02 15:16:43.196762 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.196724 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-maas-api-tls\") pod \"72bd792b-3787-4080-901b-b73ce0e7b8ea\" (UID: \"72bd792b-3787-4080-901b-b73ce0e7b8ea\") " Nov 02 15:16:43.196905 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.196882 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bd792b-3787-4080-901b-b73ce0e7b8ea-openshift-service-ca" pod "72bd792b-3787-4080-901b-b73ce0e7b8ea" (UID: "72bd792b-3787-4080-901b-b73ce0e7b8ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:43.197076 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.197060 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bd792b-3787-4080-901b-b73ce0e7b8ea-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:43.198661 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.198638 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bd792b-3787-4080-901b-b73ce0e7b8ea-kube-api-access-l4gwn" pod "72bd792b-3787-4080-901b-b73ce0e7b8ea" (UID: "72bd792b-3787-4080-901b-b73ce0e7b8ea"). InnerVolumeSpecName "kube-api-access-l4gwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:43.198746 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.198665 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-metrics-certs" pod "72bd792b-3787-4080-901b-b73ce0e7b8ea" (UID: "72bd792b-3787-4080-901b-b73ce0e7b8ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:43.198790 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.198743 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-maas-api-tls" pod "72bd792b-3787-4080-901b-b73ce0e7b8ea" (UID: "72bd792b-3787-4080-901b-b73ce0e7b8ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:43.298536 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.298493 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4gwn\" (UniqueName: \"kubernetes.io/projected/72bd792b-3787-4080-901b-b73ce0e7b8ea-kube-api-access-l4gwn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:43.298536 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.298531 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:43.298536 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.298544 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bd792b-3787-4080-901b-b73ce0e7b8ea-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:43.405572 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.405538 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:43.518586 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.518549 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6"] Nov 02 15:16:43.530588 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.530555 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-437128-584f44648f-4mxl6"] Nov 02 15:16:43.565829 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.565798 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:43.838563 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.838472 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h"] Nov 02 15:16:43.839058 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.838718 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" containerID="cri-o://0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb" gracePeriod=30 Nov 02 15:16:43.930163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:43.929934 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:44.401493 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:44.401458 5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:16:44.401668 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:44.401515 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:16:45.098645 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.098606 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm"] Nov 02 15:16:45.110546 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.110511 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jrkm"] Nov 02 15:16:45.141395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.141352 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d"] Nov 02 15:16:45.141757 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.141742 5591 cpu_manager.go:414] "containerMap: removing container" podUID="72bd792b-3787-4080-901b-b73ce0e7b8ea" containerName="maas-api" Nov 02 15:16:45.141757 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.141756 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bd792b-3787-4080-901b-b73ce0e7b8ea" containerName="maas-api" Nov 02 15:16:45.141924 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.141837 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bd792b-3787-4080-901b-b73ce0e7b8ea" containerName="maas-api" Nov 02 15:16:45.146455 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.146429 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.162401 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.162376 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:45.164169 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.164144 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:16:45.164378 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.164211 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:16:45.164378 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.164294 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:16:45.165164 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.165142 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d"] Nov 02 15:16:45.315929 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.315885 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btk59\" (UniqueName: \"kubernetes.io/projected/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-kube-api-access-btk59\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.316168 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.315955 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.316168 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.316069 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.417112 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.417076 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.417328 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.417169 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btk59\" (UniqueName: \"kubernetes.io/projected/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-kube-api-access-btk59\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.417328 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.417241 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.417887 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.417857 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.419748 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.419722 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.428292 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.428264 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ffd0af-55cc-4863-9739-15546b597534" path="/var/lib/kubelet/pods/65ffd0af-55cc-4863-9739-15546b597534/volumes" Nov 02 15:16:45.428722 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.428707 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bd792b-3787-4080-901b-b73ce0e7b8ea" path="/var/lib/kubelet/pods/72bd792b-3787-4080-901b-b73ce0e7b8ea/volumes" Nov 02 15:16:45.440252 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.440228 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btk59\" (UniqueName: \"kubernetes.io/projected/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-kube-api-access-btk59\") pod \"maas-api-revoke-sub-3d4156f1-44d9d\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.459044 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.459021 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:45.762193 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:45.762164 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d"] Nov 02 15:16:45.764836 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:16:45.764808 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8fa04b_0ff4_46ed_ac80_e710b908e95f.slice/crio-b17c212dc9f22ea4f5dacffcb9ff5bed4f1184caf938df64445b98e5bb866dfb WatchSource:0}: Error finding container b17c212dc9f22ea4f5dacffcb9ff5bed4f1184caf938df64445b98e5bb866dfb: Status 404 returned error can't find the container with id b17c212dc9f22ea4f5dacffcb9ff5bed4f1184caf938df64445b98e5bb866dfb Nov 02 15:16:46.121732 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:46.121693 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:16:46.193240 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:46.193205 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" event={"ID":"2a8fa04b-0ff4-46ed-ac80-e710b908e95f","Type":"ContainerStarted","Data":"bb2a51e231b1ac35b0a8c0bd8006fe5e5ff2e3653dae7ec8b1c8a9c9529354ef"} Nov 02 15:16:46.193411 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:46.193258 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" event={"ID":"2a8fa04b-0ff4-46ed-ac80-e710b908e95f","Type":"ContainerStarted","Data":"b17c212dc9f22ea4f5dacffcb9ff5bed4f1184caf938df64445b98e5bb866dfb"} Nov 02 15:16:46.234472 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:46.234417 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" podStartSLOduration=1.234402745 podStartE2EDuration="1.234402745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:46.232637672 +0000 UTC m=+5554.958259441" watchObservedRunningTime="2026-11-02 15:16:46.234402745 +0000 UTC m=+5554.960024511" Nov 02 15:16:47.199499 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:47.199461 5591 generic.go:356] "Generic (PLEG): container finished" podID="2a8fa04b-0ff4-46ed-ac80-e710b908e95f" containerID="bb2a51e231b1ac35b0a8c0bd8006fe5e5ff2e3653dae7ec8b1c8a9c9529354ef" exitCode=0 Nov 02 15:16:47.199939 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:47.199546 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" event={"ID":"2a8fa04b-0ff4-46ed-ac80-e710b908e95f","Type":"ContainerDied","Data":"bb2a51e231b1ac35b0a8c0bd8006fe5e5ff2e3653dae7ec8b1c8a9c9529354ef"} Nov 02 15:16:48.339637 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.339604 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:48.443902 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.443866 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-xks-ca\") pod \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " Nov 02 15:16:48.444131 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.443992 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-kube-api-access-btk59\" (UniqueName: \"kubernetes.io/projected/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-kube-api-access-btk59\") pod \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " Nov 02 15:16:48.444131 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.444038 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-openshift-ca\") pod \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\" (UID: \"2a8fa04b-0ff4-46ed-ac80-e710b908e95f\") " Nov 02 15:16:48.444453 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.444431 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-openshift-ca" pod "2a8fa04b-0ff4-46ed-ac80-e710b908e95f" (UID: "2a8fa04b-0ff4-46ed-ac80-e710b908e95f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:48.446163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.446134 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-xks-ca" pod "2a8fa04b-0ff4-46ed-ac80-e710b908e95f" (UID: "2a8fa04b-0ff4-46ed-ac80-e710b908e95f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:48.446276 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.446135 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-kube-api-access-btk59" pod "2a8fa04b-0ff4-46ed-ac80-e710b908e95f" (UID: "2a8fa04b-0ff4-46ed-ac80-e710b908e95f"). InnerVolumeSpecName "kube-api-access-btk59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:48.545710 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.545617 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btk59\" (UniqueName: \"kubernetes.io/projected/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-kube-api-access-btk59\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:48.545710 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.545658 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:48.545710 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:48.545671 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8fa04b-0ff4-46ed-ac80-e710b908e95f-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.078505 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.078479 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:16:49.210027 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.209978 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" Nov 02 15:16:49.210027 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.210011 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d" event={"ID":"2a8fa04b-0ff4-46ed-ac80-e710b908e95f","Type":"ContainerDied","Data":"b17c212dc9f22ea4f5dacffcb9ff5bed4f1184caf938df64445b98e5bb866dfb"} Nov 02 15:16:49.210333 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.210060 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17c212dc9f22ea4f5dacffcb9ff5bed4f1184caf938df64445b98e5bb866dfb" Nov 02 15:16:49.211774 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.211747 5591 generic.go:356] "Generic (PLEG): container finished" podID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerID="0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb" exitCode=0 Nov 02 15:16:49.211898 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.211785 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" event={"ID":"f0266d5e-3e55-403c-ae7e-001d058ce705","Type":"ContainerDied","Data":"0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb"} Nov 02 15:16:49.211898 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.211838 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" event={"ID":"f0266d5e-3e55-403c-ae7e-001d058ce705","Type":"ContainerDied","Data":"b76c106418e065ed9765431bb3c7fb3254feb925e58dc7aca061cab736eb5a61"} Nov 02 15:16:49.211898 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.211859 5591 scope.go:122] "RemoveContainer" containerID="0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb" Nov 02 15:16:49.211898 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.211799 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h" Nov 02 15:16:49.221187 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.221134 5591 scope.go:122] "RemoveContainer" containerID="450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d" Nov 02 15:16:49.229285 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.229262 5591 scope.go:122] "RemoveContainer" containerID="0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb" Nov 02 15:16:49.229554 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:16:49.229532 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb\": container with ID starting with 0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb not found: ID does not exist" containerID="0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb" Nov 02 15:16:49.229621 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.229561 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb"} err="failed to get container status \"0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb\": rpc error: code = NotFound desc = could not find container \"0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb\": container with ID starting with 0e0448dabe114a3e653bc3af85d0844e1849bd59fa4311c9e7b75b14753ba7fb not found: ID does not exist" Nov 02 15:16:49.229621 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.229579 5591 scope.go:122] "RemoveContainer" containerID="450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d" Nov 02 15:16:49.229828 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:16:49.229813 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d\": container with ID starting with 450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d not found: ID does not exist" containerID="450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d" Nov 02 15:16:49.229876 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.229832 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d"} err="failed to get container status \"450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d\": rpc error: code = NotFound desc = could not find container \"450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d\": container with ID starting with 450e862c1fb8a44c4009608d4e3b1684d40d83b11a62b7f461e8c474741ba81d not found: ID does not exist" Nov 02 15:16:49.252755 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.252733 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-kube-api-access-5xm24\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-kube-api-access-5xm24\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.252858 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.252781 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-data\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.252858 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.252819 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0266d5e-3e55-403c-ae7e-001d058ce705-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0266d5e-3e55-403c-ae7e-001d058ce705-istiod-ca-cert\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.252858 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.252842 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-token\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-token\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.253056 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.252885 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-envoy\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.253056 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.252906 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-certs\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.253056 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.252937 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-credential-socket\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.253056 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.252973 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-podinfo\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.253282 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253065 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-socket\") pod \"f0266d5e-3e55-403c-ae7e-001d058ce705\" (UID: \"f0266d5e-3e55-403c-ae7e-001d058ce705\") " Nov 02 15:16:49.253282 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253189 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-data" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:16:49.253400 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253276 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-certs" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:16:49.253400 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253302 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-credential-socket" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:16:49.253520 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253429 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-socket" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:16:49.253520 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253442 5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.253520 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253466 5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.253520 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253481 5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.253760 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.253736 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0266d5e-3e55-403c-ae7e-001d058ce705-istiod-ca-cert" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:49.255050 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.255025 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-envoy" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:16:49.255050 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.255031 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-kube-api-access-5xm24" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "kube-api-access-5xm24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:49.255184 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.255048 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-token" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:49.255184 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.255172 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-podinfo" pod "f0266d5e-3e55-403c-ae7e-001d058ce705" (UID: "f0266d5e-3e55-403c-ae7e-001d058ce705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:16:49.354901 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.354866 5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.355272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.354898 5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.355272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.354921 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xm24\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-kube-api-access-5xm24\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.355272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.354931 5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0266d5e-3e55-403c-ae7e-001d058ce705-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.355272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.354941 5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.355272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.354948 5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0266d5e-3e55-403c-ae7e-001d058ce705-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:16:49.551276 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.551191 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h"] Nov 02 15:16:49.563477 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:49.563440 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-437128-openshift-default-57fcdcd7df6lg4h"] Nov 02 15:16:51.429420 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:51.429376 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" path="/var/lib/kubelet/pods/f0266d5e-3e55-403c-ae7e-001d058ce705/volumes" Nov 02 15:16:56.033840 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:56.033802 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b"] Nov 02 15:16:56.045719 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:56.045683 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc46b"] Nov 02 15:16:57.326342 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:57.326302 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:16:57.429149 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:16:57.429116 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183126d4-3d0a-48af-87fb-cc0c18d6d076" path="/var/lib/kubelet/pods/183126d4-3d0a-48af-87fb-cc0c18d6d076/volumes" Nov 02 15:17:01.073071 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:17:01.073031 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:17:02.530657 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:02.530618 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:17:11.033096 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:11.033059 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr"] Nov 02 15:17:11.045022 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:11.044978 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-497nr"] Nov 02 15:17:11.428862 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:11.428821 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d" path="/var/lib/kubelet/pods/7addaaad-1e9c-4c77-8e5e-6466ce0b6b9d/volumes" Nov 02 15:17:15.265305 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:15.265270 5591 scope.go:122] "RemoveContainer" containerID="cfa99671758244b930b3bae03d20500d9369ab730f721afb59cc013b52ffe4d3" Nov 02 15:17:15.273734 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:15.273712 5591 scope.go:122] "RemoveContainer" containerID="6896fa1157fdfe9714720f43bab4b12e9ff424234fcf42870b0e0f6c1463a00e" Nov 02 15:17:15.281097 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:15.281072 5591 scope.go:122] "RemoveContainer" containerID="526fcd54d3d54eb65a98eae34b9f3081d8f61670677fff1c898e367558c7584a" Nov 02 15:17:28.028514 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:28.028423 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k"] Nov 02 15:17:28.040205 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:28.040170 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mnk4k"] Nov 02 15:17:29.429128 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:29.429091 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0a10fe-a04e-4b42-b3ad-160884777826" path="/var/lib/kubelet/pods/ed0a10fe-a04e-4b42-b3ad-160884777826/volumes" Nov 02 15:17:31.072773 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:17:31.072737 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:17:31.408842 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:31.408803 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:42.027839 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:42.027791 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh"] Nov 02 15:17:42.039628 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:42.039598 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czpwh"] Nov 02 15:17:43.428672 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:17:43.428635 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5a23c5-6c64-4ece-bb96-7e461766ebdf" path="/var/lib/kubelet/pods/ca5a23c5-6c64-4ece-bb96-7e461766ebdf/volumes" Nov 02 15:18:01.073060 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:18:01.073018 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:18:15.319994 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:15.319955 5591 scope.go:122] "RemoveContainer" containerID="e6ab304c092163e6c95e2c565d3934c173a5f6eda7659ad93d86da61b7b7be30" Nov 02 15:18:15.328618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:15.328591 5591 scope.go:122] "RemoveContainer" containerID="d04f272202db23395acf62bd3a4472f0bec83ff2aa95e693d9ba75d736752ce7" Nov 02 15:18:15.336600 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:15.336571 5591 scope.go:122] "RemoveContainer" containerID="23fa76c789c432e37bab2923c8eca3b94a07cde6cba8dbebd0082c46bd4a8503" Nov 02 15:18:15.344110 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:15.344088 5591 scope.go:122] "RemoveContainer" containerID="53fb2c8c2ba2d6296eab6433ed3c4b76e70a65aa8167d01726a9fd93e2ca9c32" Nov 02 15:18:29.631741 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:29.631703 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:18:31.073321 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:18:31.073279 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:18:31.508166 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:31.508129 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:32.745305 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.745269 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn"] Nov 02 15:18:32.745855 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.745831 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" Nov 02 15:18:32.745944 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.745857 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" Nov 02 15:18:32.745944 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.745875 5591 cpu_manager.go:414] "containerMap: removing container" podUID="2a8fa04b-0ff4-46ed-ac80-e710b908e95f" containerName="revoke-keys" Nov 02 15:18:32.745944 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.745883 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8fa04b-0ff4-46ed-ac80-e710b908e95f" containerName="revoke-keys" Nov 02 15:18:32.745944 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.745914 5591 cpu_manager.go:414] "containerMap: removing container" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" Nov 02 15:18:32.745944 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.745922 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" Nov 02 15:18:32.746266 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.746047 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" Nov 02 15:18:32.746266 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.746063 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8fa04b-0ff4-46ed-ac80-e710b908e95f" containerName="revoke-keys" Nov 02 15:18:32.746266 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.746077 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0266d5e-3e55-403c-ae7e-001d058ce705" containerName="istio-proxy" Nov 02 15:18:32.749086 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.749065 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.800371 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.800332 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:18:32.802213 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.802185 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn"] Nov 02 15:18:32.807825 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.807796 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:18:32.807825 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.807797 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:18:32.812473 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.812454 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:18:32.859784 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.859757 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.859928 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.859788 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887nr\" (UniqueName: \"kubernetes.io/projected/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-kube-api-access-887nr\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.859928 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.859903 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.960655 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.960615 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.960842 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.960696 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.960842 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.960718 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887nr\" (UniqueName: \"kubernetes.io/projected/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-kube-api-access-887nr\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.961350 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.961324 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.963108 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.963084 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:32.984295 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:32.984263 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887nr\" (UniqueName: \"kubernetes.io/projected/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-kube-api-access-887nr\") pod \"maas-api-revoke-sub-d576e68b-tpsnn\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:33.061512 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:33.061425 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:33.359783 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:33.359587 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn"] Nov 02 15:18:33.362536 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:18:33.362503 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f88bd9b_4b7f_45cf_a036_ee9eab50e08c.slice/crio-fbe289f21240bebbbd59d8f43214e8e78f189a816581c6c1c793f888f7520ff9 WatchSource:0}: Error finding container fbe289f21240bebbbd59d8f43214e8e78f189a816581c6c1c793f888f7520ff9: Status 404 returned error can't find the container with id fbe289f21240bebbbd59d8f43214e8e78f189a816581c6c1c793f888f7520ff9 Nov 02 15:18:33.786437 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:33.786387 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" event={"ID":"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c","Type":"ContainerStarted","Data":"7e4d79d3718328cbcf5ea658a611abf8103b7be2e8ce2b6c7d784e704ffa0f3d"} Nov 02 15:18:33.786841 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:33.786462 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" event={"ID":"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c","Type":"ContainerStarted","Data":"fbe289f21240bebbbd59d8f43214e8e78f189a816581c6c1c793f888f7520ff9"} Nov 02 15:18:33.826599 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:33.826543 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" podStartSLOduration=1.8265313810000001 podStartE2EDuration="1.826531381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:33.826112862 +0000 UTC m=+5662.551734632" watchObservedRunningTime="2026-11-02 15:18:33.826531381 +0000 UTC m=+5662.552153148" Nov 02 15:18:34.792570 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:34.792536 5591 generic.go:356] "Generic (PLEG): container finished" podID="0f88bd9b-4b7f-45cf-a036-ee9eab50e08c" containerID="7e4d79d3718328cbcf5ea658a611abf8103b7be2e8ce2b6c7d784e704ffa0f3d" exitCode=0 Nov 02 15:18:34.792570 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:34.792575 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" event={"ID":"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c","Type":"ContainerDied","Data":"7e4d79d3718328cbcf5ea658a611abf8103b7be2e8ce2b6c7d784e704ffa0f3d"} Nov 02 15:18:35.927364 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:35.927338 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:35.987688 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:35.987651 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-openshift-ca\") pod \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " Nov 02 15:18:35.987891 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:35.987755 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-xks-ca\") pod \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " Nov 02 15:18:35.987891 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:35.987793 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-kube-api-access-887nr\" (UniqueName: \"kubernetes.io/projected/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-kube-api-access-887nr\") pod \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\" (UID: \"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c\") " Nov 02 15:18:35.988114 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:35.988088 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-openshift-ca" pod "0f88bd9b-4b7f-45cf-a036-ee9eab50e08c" (UID: "0f88bd9b-4b7f-45cf-a036-ee9eab50e08c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:18:35.989887 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:35.989863 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-xks-ca" pod "0f88bd9b-4b7f-45cf-a036-ee9eab50e08c" (UID: "0f88bd9b-4b7f-45cf-a036-ee9eab50e08c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:18:35.990048 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:35.989864 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-kube-api-access-887nr" pod "0f88bd9b-4b7f-45cf-a036-ee9eab50e08c" (UID: "0f88bd9b-4b7f-45cf-a036-ee9eab50e08c"). InnerVolumeSpecName "kube-api-access-887nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:18:36.089475 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:36.089388 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:18:36.089475 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:36.089423 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:18:36.089475 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:36.089435 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887nr\" (UniqueName: \"kubernetes.io/projected/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c-kube-api-access-887nr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:18:36.801968 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:36.801935 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" Nov 02 15:18:36.801968 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:36.801952 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn" event={"ID":"0f88bd9b-4b7f-45cf-a036-ee9eab50e08c","Type":"ContainerDied","Data":"fbe289f21240bebbbd59d8f43214e8e78f189a816581c6c1c793f888f7520ff9"} Nov 02 15:18:36.802212 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:36.802019 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbe289f21240bebbbd59d8f43214e8e78f189a816581c6c1c793f888f7520ff9" Nov 02 15:18:37.176602 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:37.176565 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:40.027924 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:40.027875 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx"] Nov 02 15:18:40.039786 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:40.039746 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szlmx"] Nov 02 15:18:41.428380 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:41.428342 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2221d9bd-e508-49a0-91f2-4520d2f1eaf5" path="/var/lib/kubelet/pods/2221d9bd-e508-49a0-91f2-4520d2f1eaf5/volumes" Nov 02 15:18:44.027369 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:44.027332 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:18:47.725754 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.725715 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf"] Nov 02 15:18:47.726403 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.726379 5591 cpu_manager.go:414] "containerMap: removing container" podUID="0f88bd9b-4b7f-45cf-a036-ee9eab50e08c" containerName="revoke-keys" Nov 02 15:18:47.726460 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.726407 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f88bd9b-4b7f-45cf-a036-ee9eab50e08c" containerName="revoke-keys" Nov 02 15:18:47.726515 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.726501 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f88bd9b-4b7f-45cf-a036-ee9eab50e08c" containerName="revoke-keys" Nov 02 15:18:47.730417 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.730394 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:47.817756 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.817716 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\"" Nov 02 15:18:47.818077 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.818053 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:18:47.819889 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.818285 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:18:47.819889 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.818343 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:18:47.819889 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.819804 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf"] Nov 02 15:18:47.918723 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.918683 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76tc\" (UniqueName: \"kubernetes.io/projected/26a0c1e3-cd2e-4418-8244-2201bd4e984b-kube-api-access-s76tc\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:47.918913 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.918835 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:47.918966 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:47.918902 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:48.020119 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.020029 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76tc\" (UniqueName: \"kubernetes.io/projected/26a0c1e3-cd2e-4418-8244-2201bd4e984b-kube-api-access-s76tc\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:48.020285 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.020123 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:48.020285 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.020143 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:48.020893 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.020863 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:48.022560 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.022524 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:48.051006 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.048574 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76tc\" (UniqueName: \"kubernetes.io/projected/26a0c1e3-cd2e-4418-8244-2201bd4e984b-kube-api-access-s76tc\") pod \"maas-api-revoke-sub-d2bc93bc-jzcqf\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:48.075173 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.067125 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:48.484056 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.483714 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf"] Nov 02 15:18:48.904028 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:48.903217 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" event={"ID":"26a0c1e3-cd2e-4418-8244-2201bd4e984b","Type":"ContainerStarted","Data":"4d213b24a4e03785ccf95a4e7894ca1888a4b24372f03db26a1ffcb0ff3fcdee"} Nov 02 15:18:49.466177 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:49.466137 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:18:49.909626 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:49.909589 5591 generic.go:356] "Generic (PLEG): container finished" podID="26a0c1e3-cd2e-4418-8244-2201bd4e984b" containerID="5acfcecbd5e4eddebfb6352d3d2d032816b84472b548866fcf4978baede165fc" exitCode=0 Nov 02 15:18:49.910042 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:49.909648 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" event={"ID":"26a0c1e3-cd2e-4418-8244-2201bd4e984b","Type":"ContainerDied","Data":"5acfcecbd5e4eddebfb6352d3d2d032816b84472b548866fcf4978baede165fc"} Nov 02 15:18:51.037645 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.037607 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t"] Nov 02 15:18:51.041955 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.041931 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:51.049496 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.049472 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-ct64t"] Nov 02 15:18:51.056332 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.056309 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a0c1e3-cd2e-4418-8244-2201bd4e984b-kube-api-access-s76tc\" (UniqueName: \"kubernetes.io/projected/26a0c1e3-cd2e-4418-8244-2201bd4e984b-kube-api-access-s76tc\") pod \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " Nov 02 15:18:51.056453 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.056374 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-xks-ca\") pod \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " Nov 02 15:18:51.056453 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.056417 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-openshift-ca\") pod \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\" (UID: \"26a0c1e3-cd2e-4418-8244-2201bd4e984b\") " Nov 02 15:18:51.056887 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.056861 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-openshift-ca" pod "26a0c1e3-cd2e-4418-8244-2201bd4e984b" (UID: "26a0c1e3-cd2e-4418-8244-2201bd4e984b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:18:51.058305 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.058277 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a0c1e3-cd2e-4418-8244-2201bd4e984b-kube-api-access-s76tc" pod "26a0c1e3-cd2e-4418-8244-2201bd4e984b" (UID: "26a0c1e3-cd2e-4418-8244-2201bd4e984b"). InnerVolumeSpecName "kube-api-access-s76tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:18:51.058389 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.058342 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-xks-ca" pod "26a0c1e3-cd2e-4418-8244-2201bd4e984b" (UID: "26a0c1e3-cd2e-4418-8244-2201bd4e984b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:18:51.158142 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.158102 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s76tc\" (UniqueName: \"kubernetes.io/projected/26a0c1e3-cd2e-4418-8244-2201bd4e984b-kube-api-access-s76tc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:18:51.158142 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.158139 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:18:51.158142 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.158150 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a0c1e3-cd2e-4418-8244-2201bd4e984b-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:18:51.433014 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.432606 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216b9755-6d9f-4f9e-b6e6-1f621a1fccd3" path="/var/lib/kubelet/pods/216b9755-6d9f-4f9e-b6e6-1f621a1fccd3/volumes" Nov 02 15:18:51.919999 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.919959 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" Nov 02 15:18:51.920174 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.920006 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf" event={"ID":"26a0c1e3-cd2e-4418-8244-2201bd4e984b","Type":"ContainerDied","Data":"4d213b24a4e03785ccf95a4e7894ca1888a4b24372f03db26a1ffcb0ff3fcdee"} Nov 02 15:18:51.920174 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:51.920061 5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d213b24a4e03785ccf95a4e7894ca1888a4b24372f03db26a1ffcb0ff3fcdee" Nov 02 15:18:53.851449 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:18:53.851361 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:19:01.072746 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:19:01.072703 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:19:09.722526 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.722484 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm"] Nov 02 15:19:09.723103 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.722892 5591 cpu_manager.go:414] "containerMap: removing container" podUID="26a0c1e3-cd2e-4418-8244-2201bd4e984b" containerName="revoke-keys" Nov 02 15:19:09.723103 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.722907 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a0c1e3-cd2e-4418-8244-2201bd4e984b" containerName="revoke-keys" Nov 02 15:19:09.723219 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.723173 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a0c1e3-cd2e-4418-8244-2201bd4e984b" containerName="revoke-keys" Nov 02 15:19:09.726591 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.726564 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.742129 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.742076 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp"] Nov 02 15:19:09.745774 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.745750 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.753974 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.753939 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:19:09.754163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.754060 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:19:09.754163 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.753947 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:19:09.756077 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.756051 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\"" Nov 02 15:19:09.756192 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.756129 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 15:19:09.769031 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.768934 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm"] Nov 02 15:19:09.770847 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.770818 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp"] Nov 02 15:19:09.838215 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.838181 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-maas-api-tls\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.838215 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.838219 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkm8z\" (UniqueName: \"kubernetes.io/projected/0a5bbabf-833e-4334-919d-857464d2d1b2-kube-api-access-lkm8z\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.838426 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.838329 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-metrics-certs\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.838471 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.838429 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwcg\" (UniqueName: \"kubernetes.io/projected/604f07a2-a914-4540-a82c-0bb6083d0790-kube-api-access-gcwcg\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.838530 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.838515 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a5bbabf-833e-4334-919d-857464d2d1b2-metrics-certs\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.838573 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.838545 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a5bbabf-833e-4334-919d-857464d2d1b2-maas-api-tls\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.838613 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.838576 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5bbabf-833e-4334-919d-857464d2d1b2-openshift-service-ca\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.838653 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.838640 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604f07a2-a914-4540-a82c-0bb6083d0790-openshift-service-ca\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.939467 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.939420 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwcg\" (UniqueName: \"kubernetes.io/projected/604f07a2-a914-4540-a82c-0bb6083d0790-kube-api-access-gcwcg\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.939691 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.939486 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a5bbabf-833e-4334-919d-857464d2d1b2-metrics-certs\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.939691 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.939515 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a5bbabf-833e-4334-919d-857464d2d1b2-maas-api-tls\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.939691 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.939548 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5bbabf-833e-4334-919d-857464d2d1b2-openshift-service-ca\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.939691 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.939571 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604f07a2-a914-4540-a82c-0bb6083d0790-openshift-service-ca\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.939691 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.939624 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-maas-api-tls\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.939691 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.939674 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkm8z\" (UniqueName: \"kubernetes.io/projected/0a5bbabf-833e-4334-919d-857464d2d1b2-kube-api-access-lkm8z\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.940213 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.939711 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-metrics-certs\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.940537 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.940508 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5bbabf-833e-4334-919d-857464d2d1b2-openshift-service-ca\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.940654 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.940567 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604f07a2-a914-4540-a82c-0bb6083d0790-openshift-service-ca\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.942330 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.942302 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a5bbabf-833e-4334-919d-857464d2d1b2-metrics-certs\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.949387 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.942365 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-maas-api-tls\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.949387 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.942824 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-metrics-certs\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.949387 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.943086 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a5bbabf-833e-4334-919d-857464d2d1b2-maas-api-tls\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:09.963011 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.962948 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwcg\" (UniqueName: \"kubernetes.io/projected/604f07a2-a914-4540-a82c-0bb6083d0790-kube-api-access-gcwcg\") pod \"maas-api-c77ff87df-6fnjm\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:09.963149 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:09.963088 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkm8z\" (UniqueName: \"kubernetes.io/projected/0a5bbabf-833e-4334-919d-857464d2d1b2-kube-api-access-lkm8z\") pod \"maas-api-77c79c44b5-28ljp\" (UID: \"0a5bbabf-833e-4334-919d-857464d2d1b2\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:10.042316 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:10.042230 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:10.063101 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:10.063069 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:10.347529 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:10.347305 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm"] Nov 02 15:19:10.350520 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:19:10.350488 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604f07a2_a914_4540_a82c_0bb6083d0790.slice/crio-e77d354f999b42ffa4007d5a74bd68c3d72330710176f00432172f27675d5df5 WatchSource:0}: Error finding container e77d354f999b42ffa4007d5a74bd68c3d72330710176f00432172f27675d5df5: Status 404 returned error can't find the container with id e77d354f999b42ffa4007d5a74bd68c3d72330710176f00432172f27675d5df5 Nov 02 15:19:10.390071 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:10.389908 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp"] Nov 02 15:19:10.392438 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:19:10.392409 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5bbabf_833e_4334_919d_857464d2d1b2.slice/crio-948f84ffb0f7745e31efd7f7aa9daa40ea521c4daca5eab767f0056615aaad48 WatchSource:0}: Error finding container 948f84ffb0f7745e31efd7f7aa9daa40ea521c4daca5eab767f0056615aaad48: Status 404 returned error can't find the container with id 948f84ffb0f7745e31efd7f7aa9daa40ea521c4daca5eab767f0056615aaad48 Nov 02 15:19:11.011018 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:11.010900 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" event={"ID":"0a5bbabf-833e-4334-919d-857464d2d1b2","Type":"ContainerStarted","Data":"f4b8b69defc396061c7c0189b87a184deb0f3da1932a3b68f65ee8f9d2fd2f36"} Nov 02 15:19:11.011018 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:11.010973 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:11.011509 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:11.011025 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" event={"ID":"0a5bbabf-833e-4334-919d-857464d2d1b2","Type":"ContainerStarted","Data":"948f84ffb0f7745e31efd7f7aa9daa40ea521c4daca5eab767f0056615aaad48"} Nov 02 15:19:11.012636 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:11.012601 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" event={"ID":"604f07a2-a914-4540-a82c-0bb6083d0790","Type":"ContainerStarted","Data":"a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532"} Nov 02 15:19:11.012763 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:11.012642 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" event={"ID":"604f07a2-a914-4540-a82c-0bb6083d0790","Type":"ContainerStarted","Data":"e77d354f999b42ffa4007d5a74bd68c3d72330710176f00432172f27675d5df5"} Nov 02 15:19:11.012763 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:11.012684 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:11.056348 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:11.056295 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" podStartSLOduration=1.689896055 podStartE2EDuration="2.05628121s" totalImagesPullingTime="366.385155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:11.054601798 +0000 UTC m=+5699.780223568" watchObservedRunningTime="2026-11-02 15:19:11.05628121 +0000 UTC m=+5699.781902980" Nov 02 15:19:11.097823 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:11.097760 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" podStartSLOduration=1.670098619 podStartE2EDuration="2.097740466s" totalImagesPullingTime="427.641847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:11.09565135 +0000 UTC m=+5699.821273121" watchObservedRunningTime="2026-11-02 15:19:11.097740466 +0000 UTC m=+5699.823362237" Nov 02 15:19:12.481058 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:12.481030 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:19:12.492272 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:12.492249 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:19:15.373609 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:15.373576 5591 scope.go:122] "RemoveContainer" containerID="3b3217a7ef01cbb58ac64693b3839845b62bac6fc2a6b5c0b1eef6b0403389ae" Nov 02 15:19:15.383118 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:15.383094 5591 scope.go:122] "RemoveContainer" containerID="234af82e33810b558c5c27505842bcb2eecb4fa26ccb7a4a9cbbf96eae8acc7c" Nov 02 15:19:15.392852 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:15.392831 5591 scope.go:122] "RemoveContainer" containerID="2e4b318fa4c4f6d85020523145aa5640bd0248add043243e7d592a049d8124c7" Nov 02 15:19:15.400915 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:15.400891 5591 scope.go:122] "RemoveContainer" containerID="e4a4fbbbac1fd492b7416a33a6d862331e8ff5493128f07e14f96d40d95ac59a" Nov 02 15:19:17.023515 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.023479 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-28ljp" Nov 02 15:19:17.023962 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.023833 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:17.121112 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.121072 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm"] Nov 02 15:19:17.121350 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.121323 5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" podUID="604f07a2-a914-4540-a82c-0bb6083d0790" containerName="maas-api" containerID="cri-o://a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532" gracePeriod=30 Nov 02 15:19:17.372686 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.372660 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:17.513472 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.513434 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604f07a2-a914-4540-a82c-0bb6083d0790-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604f07a2-a914-4540-a82c-0bb6083d0790-openshift-service-ca\") pod \"604f07a2-a914-4540-a82c-0bb6083d0790\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " Nov 02 15:19:17.513472 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.513474 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604f07a2-a914-4540-a82c-0bb6083d0790-kube-api-access-gcwcg\" (UniqueName: \"kubernetes.io/projected/604f07a2-a914-4540-a82c-0bb6083d0790-kube-api-access-gcwcg\") pod \"604f07a2-a914-4540-a82c-0bb6083d0790\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " Nov 02 15:19:17.513724 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.513517 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-metrics-certs\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-metrics-certs\") pod \"604f07a2-a914-4540-a82c-0bb6083d0790\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " Nov 02 15:19:17.513724 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.513692 5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-maas-api-tls\") pod \"604f07a2-a914-4540-a82c-0bb6083d0790\" (UID: \"604f07a2-a914-4540-a82c-0bb6083d0790\") " Nov 02 15:19:17.513868 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.513835 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604f07a2-a914-4540-a82c-0bb6083d0790-openshift-service-ca" pod "604f07a2-a914-4540-a82c-0bb6083d0790" (UID: "604f07a2-a914-4540-a82c-0bb6083d0790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:19:17.514111 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.514094 5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604f07a2-a914-4540-a82c-0bb6083d0790-openshift-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:19:17.515793 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.515765 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-metrics-certs" pod "604f07a2-a914-4540-a82c-0bb6083d0790" (UID: "604f07a2-a914-4540-a82c-0bb6083d0790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:19:17.515897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.515778 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-maas-api-tls" pod "604f07a2-a914-4540-a82c-0bb6083d0790" (UID: "604f07a2-a914-4540-a82c-0bb6083d0790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:19:17.515897 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.515828 5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604f07a2-a914-4540-a82c-0bb6083d0790-kube-api-access-gcwcg" pod "604f07a2-a914-4540-a82c-0bb6083d0790" (UID: "604f07a2-a914-4540-a82c-0bb6083d0790"). InnerVolumeSpecName "kube-api-access-gcwcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:19:17.614953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.614872 5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-maas-api-tls\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:19:17.614953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.614899 5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwcg\" (UniqueName: \"kubernetes.io/projected/604f07a2-a914-4540-a82c-0bb6083d0790-kube-api-access-gcwcg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:19:17.614953 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:17.614908 5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604f07a2-a914-4540-a82c-0bb6083d0790-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\"" Nov 02 15:19:18.047769 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.047729 5591 generic.go:356] "Generic (PLEG): container finished" podID="604f07a2-a914-4540-a82c-0bb6083d0790" containerID="a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532" exitCode=0 Nov 02 15:19:18.048200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.047783 5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" Nov 02 15:19:18.048200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.047788 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" event={"ID":"604f07a2-a914-4540-a82c-0bb6083d0790","Type":"ContainerDied","Data":"a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532"} Nov 02 15:19:18.048200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.047905 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm" event={"ID":"604f07a2-a914-4540-a82c-0bb6083d0790","Type":"ContainerDied","Data":"e77d354f999b42ffa4007d5a74bd68c3d72330710176f00432172f27675d5df5"} Nov 02 15:19:18.048200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.047951 5591 scope.go:122] "RemoveContainer" containerID="a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532" Nov 02 15:19:18.057751 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.057729 5591 scope.go:122] "RemoveContainer" containerID="a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532" Nov 02 15:19:18.058177 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:19:18.058155 5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532\": container with ID starting with a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532 not found: ID does not exist" containerID="a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532" Nov 02 15:19:18.058230 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.058186 5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532"} err="failed to get container status \"a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532\": rpc error: code = NotFound desc = could not find container \"a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532\": container with ID starting with a7d8b370159d44c81cd81d7dcfbb8334a1663ea09d4cc192549d6a576ee94532 not found: ID does not exist" Nov 02 15:19:18.099524 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.099491 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm"] Nov 02 15:19:18.111473 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:18.111443 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-6fnjm"] Nov 02 15:19:19.428744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:19.428709 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604f07a2-a914-4540-a82c-0bb6083d0790" path="/var/lib/kubelet/pods/604f07a2-a914-4540-a82c-0bb6083d0790/volumes" Nov 02 15:19:20.627180 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:20.627144 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:19:31.073549 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:19:31.073508 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:19:31.637107 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:31.637070 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:49.764587 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:49.764547 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:19:54.354167 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:54.354126 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:19:59.664993 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:19:59.664941 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:01.072862 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:20:01.072822 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:20:01.596200 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:01.596165 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:11.418334 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:11.418290 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:20:15.436019 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:15.435964 5591 scope.go:122] "RemoveContainer" containerID="4a13f1b6777ffec8735bcad680408d076eac0786e88bbb98cfa660dfe84655b9" Nov 02 15:20:15.444311 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:15.444289 5591 scope.go:122] "RemoveContainer" containerID="f51d355ff8eb099b9f1c85a1e090ada8d4110a09c152e56140617737d7356e4f" Nov 02 15:20:15.839399 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:15.839304 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:20:21.762549 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:21.762494 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:21.956772 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:21.956738 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:22.540012 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:22.539079 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qqfsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:20:22.540012 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:22.539196 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qqfsh" podUID="ad6d5b6e-e351-4020-81b4-98bb9d0a013a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:22.968584 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:22.968545 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:20:31.073321 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:20:31.073281 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:20:31.744689 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:31.744655 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:38.085029 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:38.084975 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:42.142164 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:42.142124 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:20:52.326372 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:52.326335 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:20:56.947896 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:20:56.947858 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:21:01.075740 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:21:01.075695 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:21:05.843300 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:05.843267 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:21:15.469187 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:15.469153 5591 scope.go:122] "RemoveContainer" containerID="faab284ebd84856af358b335905def8c87accf768848793d8494454adc8b32eb" Nov 02 15:21:22.318818 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:22.318776 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:23.054534 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:23.054498 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:24.032717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:24.032681 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:21:31.072817 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:21:31.072774 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:21:31.873476 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:31.873440 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:34.251516 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:34.251468 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:21:39.673622 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:39.673582 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:21:49.028533 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:49.028492 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d"] Nov 02 15:21:49.040309 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:49.040275 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44d9d"] Nov 02 15:21:49.444346 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:49.444298 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8fa04b-0ff4-46ed-ac80-e710b908e95f" path="/var/lib/kubelet/pods/2a8fa04b-0ff4-46ed-ac80-e710b908e95f/volumes" Nov 02 15:21:52.216038 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:21:52.215973 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:22:01.073140 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:22:01.073051 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:22:10.349472 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:22:10.349437 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:22:31.074227 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:22:31.074184 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:22:31.996695 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:22:31.996657 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:32.303818 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:22:32.303735 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:01.073071 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:23:01.073027 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:23:14.813644 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:14.813599 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:15.511694 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:15.511657 5591 scope.go:122] "RemoveContainer" containerID="bb2a51e231b1ac35b0a8c0bd8006fe5e5ff2e3653dae7ec8b1c8a9c9529354ef" Nov 02 15:23:31.073029 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:23:31.072975 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:23:32.128221 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:32.128188 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:32.432210 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:32.432175 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:35.841008 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:35.839133 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:23:35.841008 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:35.839217 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:36.030495 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:36.030451 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn"] Nov 02 15:23:36.043476 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:36.043433 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tpsnn"] Nov 02 15:23:37.430812 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:37.430773 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f88bd9b-4b7f-45cf-a036-ee9eab50e08c" path="/var/lib/kubelet/pods/0f88bd9b-4b7f-45cf-a036-ee9eab50e08c/volumes" Nov 02 15:23:52.040245 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:52.040198 5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf"] Nov 02 15:23:52.052025 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:52.051996 5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jzcqf"] Nov 02 15:23:53.430128 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:53.430091 5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a0c1e3-cd2e-4418-8244-2201bd4e984b" path="/var/lib/kubelet/pods/26a0c1e3-cd2e-4418-8244-2201bd4e984b/volumes" Nov 02 15:23:55.264632 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:23:55.264498 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:01.072480 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:24:01.072430 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:24:03.920132 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:03.920095 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4zfwv"] Nov 02 15:24:05.849010 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:05.848082 5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-xqg2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:24:05.849010 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:05.848169 5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xqg2g" podUID="cf93ec7a-9a28-4e39-b0e2-79e786e431de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:12.509604 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:12.509567 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:24:12.521682 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:12.521655 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mq8vn_52eda2b9-8258-4f7e-bc72-6ad9f6f1167a/ovn-acl-logging/0.log" Nov 02 15:24:15.901223 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:15.901187 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:31.072429 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:24:31.072389 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:24:32.232834 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:32.232800 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:40.675003 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:40.674947 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:40.845024 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:40.844966 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:46.713284 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:46.713238 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b7f5f45bd-8zr8v_a658ad5a-9b96-42c1-af92-c22f12558516/manager/0.log" Nov 02 15:24:47.730403 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:47.730366 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f746cf4d4-kpf4p_90e3c4f6-049a-49fe-88e5-b0172072ea87/manager/0.log" Nov 02 15:24:49.361513 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:49.361481 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77c79c44b5-28ljp_0a5bbabf-833e-4334-919d-857464d2d1b2/maas-api/0.log" Nov 02 15:24:55.256094 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:55.256046 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sj286_d529065b-33fe-4288-b543-a51b5de16f88/manager/0.log" Nov 02 15:24:55.910745 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:55.910710 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4zfwv_1c4e7da5-37b7-45bd-9840-6c87ab416429/limitador/0.log" Nov 02 15:24:56.091791 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:56.091750 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-sxzjh_c7a64f22-0110-479d-900e-b27ee83b6985/manager/0.log" Nov 02 15:24:57.290146 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:57.290114 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tnmvk_f3247735-5d35-43ac-a9cd-48c53d43ab19/discovery/0.log" Nov 02 15:24:57.509170 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:57.509118 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd64f76dd-9f2pc_a0c62419-de89-4af5-951b-52a33f4ac5b1/kube-auth-proxy/0.log" Nov 02 15:24:59.105215 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:59.105181 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qqfsh_ad6d5b6e-e351-4020-81b4-98bb9d0a013a/istio-proxy/0.log" Nov 02 15:24:59.511176 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:59.511141 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-wrbs7_dae59a45-652a-46dc-a5e7-c983d14b2b7e/istio-proxy/0.log" Nov 02 15:24:59.749647 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:24:59.749605 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-xqg2g_cf93ec7a-9a28-4e39-b0e2-79e786e431de/istio-proxy/0.log" Nov 02 15:25:00.155130 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:00.155094 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-84dc7c97f-5mmbq_f88a9ba7-3c6d-47dd-aae0-da3b997533e8/payload-pre-processing/0.log" Nov 02 15:25:01.073097 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:25:01.073055 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:25:11.599442 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:11.599408 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-lqsf2_13ac7768-2c53-40ab-af16-1d08fa52e223/main/0.log" Nov 02 15:25:11.998954 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:11.998923 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xppxm_597f5679-40fd-4228-9182-86f23da49976/main/0.log" Nov 02 15:25:12.414343 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:12.414312 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gssb6_5fc319d8-bcb7-4352-b711-f72f71b56811/main/0.log" Nov 02 15:25:12.606835 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:12.606807 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6ccj9_702ef238-5f96-4a86-b35e-9002f0738f78/main/0.log" Nov 02 15:25:15.541798 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:15.541760 5591 scope.go:122] "RemoveContainer" containerID="7e4d79d3718328cbcf5ea658a611abf8103b7be2e8ce2b6c7d784e704ffa0f3d" Nov 02 15:25:15.550339 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:15.550316 5591 scope.go:122] "RemoveContainer" containerID="5acfcecbd5e4eddebfb6352d3d2d032816b84472b548866fcf4978baede165fc" Nov 02 15:25:31.071795 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:25:31.071753 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:25:32.351996 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:32.351954 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:34.083591 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.083558 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:34.088684 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.088646 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nbgqs/must-gather-s2m8k"] Nov 02 15:25:34.089123 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.089105 5591 cpu_manager.go:414] "containerMap: removing container" podUID="604f07a2-a914-4540-a82c-0bb6083d0790" containerName="maas-api" Nov 02 15:25:34.089193 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.089124 5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604f07a2-a914-4540-a82c-0bb6083d0790" containerName="maas-api" Nov 02 15:25:34.089237 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.089209 5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="604f07a2-a914-4540-a82c-0bb6083d0790" containerName="maas-api" Nov 02 15:25:34.092290 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.092274 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbgqs/must-gather-s2m8k" Nov 02 15:25:34.106958 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.106938 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nbgqs\"/\"default-dockercfg-h8fq5\"" Nov 02 15:25:34.119102 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.119075 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nbgqs\"/\"openshift-service-ca.crt\"" Nov 02 15:25:34.119102 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.119093 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nbgqs\"/\"kube-root-ca.crt\"" Nov 02 15:25:34.130407 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.130380 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbgqs/must-gather-s2m8k"] Nov 02 15:25:34.140990 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.140942 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a5479f6-73c9-4c67-8b33-339891604e20-must-gather-output\") pod \"must-gather-s2m8k\" (UID: \"6a5479f6-73c9-4c67-8b33-339891604e20\") " pod="openshift-must-gather-nbgqs/must-gather-s2m8k" Nov 02 15:25:34.141107 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.141074 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskv9\" (UniqueName: \"kubernetes.io/projected/6a5479f6-73c9-4c67-8b33-339891604e20-kube-api-access-gskv9\") pod \"must-gather-s2m8k\" (UID: \"6a5479f6-73c9-4c67-8b33-339891604e20\") " pod="openshift-must-gather-nbgqs/must-gather-s2m8k" Nov 02 15:25:34.242045 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.242004 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a5479f6-73c9-4c67-8b33-339891604e20-must-gather-output\") pod \"must-gather-s2m8k\" (UID: \"6a5479f6-73c9-4c67-8b33-339891604e20\") " pod="openshift-must-gather-nbgqs/must-gather-s2m8k" Nov 02 15:25:34.242277 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.242052 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskv9\" (UniqueName: \"kubernetes.io/projected/6a5479f6-73c9-4c67-8b33-339891604e20-kube-api-access-gskv9\") pod \"must-gather-s2m8k\" (UID: \"6a5479f6-73c9-4c67-8b33-339891604e20\") " pod="openshift-must-gather-nbgqs/must-gather-s2m8k" Nov 02 15:25:34.242445 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.242419 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6a5479f6-73c9-4c67-8b33-339891604e20-must-gather-output\") pod \"must-gather-s2m8k\" (UID: \"6a5479f6-73c9-4c67-8b33-339891604e20\") " pod="openshift-must-gather-nbgqs/must-gather-s2m8k" Nov 02 15:25:34.266280 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.266252 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskv9\" (UniqueName: \"kubernetes.io/projected/6a5479f6-73c9-4c67-8b33-339891604e20-kube-api-access-gskv9\") pod \"must-gather-s2m8k\" (UID: \"6a5479f6-73c9-4c67-8b33-339891604e20\") " pod="openshift-must-gather-nbgqs/must-gather-s2m8k" Nov 02 15:25:34.288174 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.288148 5591 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:34.413759 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.413734 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbgqs/must-gather-s2m8k" Nov 02 15:25:34.715366 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.715292 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbgqs/must-gather-s2m8k"] Nov 02 15:25:34.718474 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:25:34.718448 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a5479f6_73c9_4c67_8b33_339891604e20.slice/crio-13a5d54096242a4f754d60dcca3cd465d34911ec5c5ebf28f0ff4c196f9cf0d6 WatchSource:0}: Error finding container 13a5d54096242a4f754d60dcca3cd465d34911ec5c5ebf28f0ff4c196f9cf0d6: Status 404 returned error can't find the container with id 13a5d54096242a4f754d60dcca3cd465d34911ec5c5ebf28f0ff4c196f9cf0d6 Nov 02 15:25:34.720295 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:34.720278 5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:25:35.067908 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:35.067818 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbgqs/must-gather-s2m8k" event={"ID":"6a5479f6-73c9-4c67-8b33-339891604e20","Type":"ContainerStarted","Data":"13a5d54096242a4f754d60dcca3cd465d34911ec5c5ebf28f0ff4c196f9cf0d6"} Nov 02 15:25:38.594145 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:38.594107 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-scx6x_a14e8823-8f98-40c4-b20d-dfea320c8304/global-pull-secret-syncer/0.log" Nov 02 15:25:39.535704 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:39.535670 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8hhsr_b34d67be-03c6-42d2-8e6f-d5936d7d940b/konnectivity-agent/0.log" Nov 02 15:25:40.171025 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:40.170977 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-236.ec2.internal_e82d0dbde7394ec25631244e1f1d5cd9/haproxy/0.log" Nov 02 15:25:41.102376 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:41.102338 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbgqs/must-gather-s2m8k" event={"ID":"6a5479f6-73c9-4c67-8b33-339891604e20","Type":"ContainerStarted","Data":"3dfd83f599b8eb592bbffa2c9c6c5d7d10fcbbf8811acd5d0c0916c20d4bcd51"} Nov 02 15:25:41.102580 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:41.102387 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbgqs/must-gather-s2m8k" event={"ID":"6a5479f6-73c9-4c67-8b33-339891604e20","Type":"ContainerStarted","Data":"fce5a711732e73238302c557f8843919444b77f6ddab1dd8228821b7965b9120"} Nov 02 15:25:41.145002 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:41.144942 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nbgqs/must-gather-s2m8k" podStartSLOduration=1.754193827 podStartE2EDuration="7.144929899s" totalImagesPullingTime="5.390736072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:41.144834531 +0000 UTC m=+6089.870456301" watchObservedRunningTime="2026-11-02 15:25:41.144929899 +0000 UTC m=+6089.870551712" Nov 02 15:25:42.039463 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:42.039427 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b7f5f45bd-8zr8v_a658ad5a-9b96-42c1-af92-c22f12558516/manager/0.log" Nov 02 15:25:43.095177 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:43.095139 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f746cf4d4-kpf4p_90e3c4f6-049a-49fe-88e5-b0172072ea87/manager/0.log" Nov 02 15:25:44.612770 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:44.612702 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b7f5f45bd-8zr8v_a658ad5a-9b96-42c1-af92-c22f12558516/manager/0.log" Nov 02 15:25:45.679706 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:45.679665 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f746cf4d4-kpf4p_90e3c4f6-049a-49fe-88e5-b0172072ea87/manager/0.log" Nov 02 15:25:46.155250 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:46.155219 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sj286_d529065b-33fe-4288-b543-a51b5de16f88/manager/0.log" Nov 02 15:25:46.570611 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:46.570536 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4zfwv_1c4e7da5-37b7-45bd-9840-6c87ab416429/limitador/0.log" Nov 02 15:25:46.766701 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:46.766669 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-sxzjh_c7a64f22-0110-479d-900e-b27ee83b6985/manager/0.log" Nov 02 15:25:47.198090 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:47.198056 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b7f5f45bd-8zr8v_a658ad5a-9b96-42c1-af92-c22f12558516/manager/0.log" Nov 02 15:25:48.276824 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.276787 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f746cf4d4-kpf4p_90e3c4f6-049a-49fe-88e5-b0172072ea87/manager/0.log" Nov 02 15:25:48.379283 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.379240 5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz"] Nov 02 15:25:48.397793 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.397758 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.401668 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.401632 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz"] Nov 02 15:25:48.484872 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.484822 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-lib-modules\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.485094 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.484914 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-sys\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.485094 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.484953 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-proc\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.485094 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.484978 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wptc\" (UniqueName: \"kubernetes.io/projected/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-kube-api-access-5wptc\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.485230 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.485186 5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-podres\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.489976 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.489955 5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gnwq9\"/\"default-dockercfg-8v5rb\"" Nov 02 15:25:48.491307 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.491290 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnwq9\"/\"kube-root-ca.crt\"" Nov 02 15:25:48.491368 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.491329 5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnwq9\"/\"openshift-service-ca.crt\"" Nov 02 15:25:48.586263 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586172 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-podres\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.586263 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586240 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-lib-modules\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.586263 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586262 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-sys\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.586578 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586283 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-proc\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.586578 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586305 5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wptc\" (UniqueName: \"kubernetes.io/projected/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-kube-api-access-5wptc\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.586578 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586315 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-podres\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.586578 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586364 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-lib-modules\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.586578 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586371 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-sys\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.586578 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.586393 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-proc\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.609618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.609583 5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wptc\" (UniqueName: \"kubernetes.io/projected/23a7f2af-6f65-4fc3-ab17-7ed87bc60c87-kube-api-access-5wptc\") pod \"perf-node-gather-daemonset-6d7kz\" (UID: \"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:48.712317 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:48.712278 5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:49.010667 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:49.010628 5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz"] Nov 02 15:25:49.016549 ip-10-0-140-236 kubenswrapper[5591]: W1102 15:25:49.016509 5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23a7f2af_6f65_4fc3_ab17_7ed87bc60c87.slice/crio-acc7a0b836ed6c0ae0cb631cae20e8602c2a64afdb93fd9b802fe1a969ed0bf2 WatchSource:0}: Error finding container acc7a0b836ed6c0ae0cb631cae20e8602c2a64afdb93fd9b802fe1a969ed0bf2: Status 404 returned error can't find the container with id acc7a0b836ed6c0ae0cb631cae20e8602c2a64afdb93fd9b802fe1a969ed0bf2 Nov 02 15:25:49.144293 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:49.144257 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" event={"ID":"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87","Type":"ContainerStarted","Data":"acc7a0b836ed6c0ae0cb631cae20e8602c2a64afdb93fd9b802fe1a969ed0bf2"} Nov 02 15:25:49.546559 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:49.546506 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-vdjd6_d5655095-e60c-40a9-9f0d-510e96cdd667/monitoring-plugin/0.log" Nov 02 15:25:50.150063 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:50.150026 5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" event={"ID":"23a7f2af-6f65-4fc3-ab17-7ed87bc60c87","Type":"ContainerStarted","Data":"8cc3d7e5f53b89104a1de50ac20fd7d86cf1c4d8386e53d10259c9efd6e7aa79"} Nov 02 15:25:50.150456 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:50.150121 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:50.190744 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:50.190685 5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" podStartSLOduration=2.190667558 podStartE2EDuration="2.190667558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 15:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:50.189902702 +0000 UTC m=+6098.915524473" watchObservedRunningTime="2026-11-02 15:25:50.190667558 +0000 UTC m=+6098.916289329" Nov 02 15:25:50.364643 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:50.364613 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sjj7t_d8e273bc-9f51-4993-86c2-7e02c3b88c7d/node-exporter/0.log" Nov 02 15:25:50.490501 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:50.490414 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sjj7t_d8e273bc-9f51-4993-86c2-7e02c3b88c7d/kube-rbac-proxy/0.log" Nov 02 15:25:50.610766 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:50.610723 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sjj7t_d8e273bc-9f51-4993-86c2-7e02c3b88c7d/init-textfile/0.log" Nov 02 15:25:51.520297 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:51.520265 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tnmvk_f3247735-5d35-43ac-a9cd-48c53d43ab19/discovery/0.log" Nov 02 15:25:51.556423 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:51.556386 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd64f76dd-9f2pc_a0c62419-de89-4af5-951b-52a33f4ac5b1/kube-auth-proxy/0.log" Nov 02 15:25:51.906856 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:51.906806 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jpn6p_4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/prometheus-operator/0.log" Nov 02 15:25:52.020803 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.020748 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jpn6p_4c6b7dd8-15b5-47b4-8fe7-b8a6d106cf35/kube-rbac-proxy/0.log" Nov 02 15:25:52.091626 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.091585 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qqfsh_ad6d5b6e-e351-4020-81b4-98bb9d0a013a/istio-proxy/0.log" Nov 02 15:25:52.235603 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.235519 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-wrbs7_dae59a45-652a-46dc-a5e7-c983d14b2b7e/istio-proxy/0.log" Nov 02 15:25:52.265718 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.265662 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69ff98d85d-s4mqk_3f56563e-0aeb-46ab-9f7f-ae0f6409ec33/telemeter-client/0.log" Nov 02 15:25:52.302079 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.302054 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-xqg2g_cf93ec7a-9a28-4e39-b0e2-79e786e431de/istio-proxy/0.log" Nov 02 15:25:52.384704 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.384678 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69ff98d85d-s4mqk_3f56563e-0aeb-46ab-9f7f-ae0f6409ec33/reload/0.log" Nov 02 15:25:52.433202 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.433172 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-84dc7c97f-5mmbq_f88a9ba7-3c6d-47dd-aae0-da3b997533e8/payload-pre-processing/0.log" Nov 02 15:25:52.504467 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.504359 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69ff98d85d-s4mqk_3f56563e-0aeb-46ab-9f7f-ae0f6409ec33/kube-rbac-proxy/0.log" Nov 02 15:25:52.630588 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.629501 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c6896987-j2hml_98a7d07b-33d6-4778-8bb0-7c277468d407/thanos-query/0.log" Nov 02 15:25:52.752765 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.752733 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c6896987-j2hml_98a7d07b-33d6-4778-8bb0-7c277468d407/kube-rbac-proxy-web/0.log" Nov 02 15:25:52.873395 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.873296 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c6896987-j2hml_98a7d07b-33d6-4778-8bb0-7c277468d407/kube-rbac-proxy/0.log" Nov 02 15:25:52.992260 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:52.992228 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c6896987-j2hml_98a7d07b-33d6-4778-8bb0-7c277468d407/prom-label-proxy/0.log" Nov 02 15:25:53.114027 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:53.113954 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c6896987-j2hml_98a7d07b-33d6-4778-8bb0-7c277468d407/kube-rbac-proxy-rules/0.log" Nov 02 15:25:53.233618 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:53.233590 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c6896987-j2hml_98a7d07b-33d6-4778-8bb0-7c277468d407/kube-rbac-proxy-metrics/0.log" Nov 02 15:25:56.166669 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:56.166633 5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-6d7kz" Nov 02 15:25:56.753523 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:56.753490 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sj286_d529065b-33fe-4288-b543-a51b5de16f88/manager/0.log" Nov 02 15:25:56.962415 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:56.962383 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4zfwv_1c4e7da5-37b7-45bd-9840-6c87ab416429/limitador/0.log" Nov 02 15:25:57.095668 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:25:57.095593 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-sxzjh_c7a64f22-0110-479d-900e-b27ee83b6985/manager/0.log" Nov 02 15:26:00.017115 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:00.017081 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6bll8_257163a1-1ead-438e-94c2-ca672bdcaa17/dns/0.log" Nov 02 15:26:00.212802 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:00.212744 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6bll8_257163a1-1ead-438e-94c2-ca672bdcaa17/kube-rbac-proxy/0.log" Nov 02 15:26:00.950521 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:00.950492 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9lm6_790cdff4-1780-4548-a327-e7b7df474347/dns-node-resolver/0.log" Nov 02 15:26:01.090865 ip-10-0-140-236 kubenswrapper[5591]: E1102 15:26:01.090823 5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kgf" containerName="extractor" Nov 02 15:26:02.470368 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:02.470320 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9d7n9_5291323c-ea92-499c-b246-a4f692f3f8c5/node-ca/0.log" Nov 02 15:26:02.626888 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:02.626858 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tnmvk_f3247735-5d35-43ac-a9cd-48c53d43ab19/discovery/0.log" Nov 02 15:26:02.661692 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:02.661655 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd64f76dd-9f2pc_a0c62419-de89-4af5-951b-52a33f4ac5b1/kube-auth-proxy/0.log" Nov 02 15:26:03.213220 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:03.213180 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qqfsh_ad6d5b6e-e351-4020-81b4-98bb9d0a013a/istio-proxy/0.log" Nov 02 15:26:03.362246 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:03.362211 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-wrbs7_dae59a45-652a-46dc-a5e7-c983d14b2b7e/istio-proxy/0.log" Nov 02 15:26:03.435353 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:03.435314 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-xqg2g_cf93ec7a-9a28-4e39-b0e2-79e786e431de/istio-proxy/0.log" Nov 02 15:26:03.593717 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:03.593645 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-84dc7c97f-5mmbq_f88a9ba7-3c6d-47dd-aae0-da3b997533e8/payload-pre-processing/0.log" Nov 02 15:26:05.682005 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:05.676406 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tnmvk_f3247735-5d35-43ac-a9cd-48c53d43ab19/discovery/0.log" Nov 02 15:26:06.385494 ip-10-0-140-236 kubenswrapper[5591]: I1102 15:26:06.385463 5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd64f76dd-9f2pc_a0c62419-de89-4af5-951b-52a33f4ac5b1/kube-auth-proxy/0.log"